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Preface to First Edition 


Linear programming (LP) founded by Dantzig (1948-1951b) might be one of the 
most well-known and widely used mathematical tools in the world. As a branch 
of optimization, it serves as the most important cornerstone of operations research, 
decision science, and management science. 

This branch emerged when the American mathematician George B. Dantzig 
created the LP model and the simplex method in 1947. The computer, emerging 
around the same period, propelled the development of LP and the simplex method 
toward practical applications. As a basic branch, LP orchestrated the birth of 
a number of new branches, such as nonlinear programming, network flow and 
combinatorial optimization, stochastic programming, integer programming, and 
complementary theory, etc., and activated the whole field of operations research. 

A prominent feature of LP is given by its broad applications. Closely related 
to LP, many individuals have made pioneering contributions in their respective 
fields. In the field of economics, in particular, Russian-American economist Wassily 
Leontief took the 1973 Nobel Economic Prize for his epoch-making contribution 
to quantitative analysis of economic activities. The academician L. V. Kantorovich 
of the former Soviet Academy of Science and American economist Professor T. C. 
Koopmans won the 1975 Nobel Prize for their optimal allocation theory of resources 
using LP. The same prize was also given to Professors K. Arrow, P. Samuelson, H. 
Simon, and L. Herwricz, several decades later when they paid close attention to LP 
at the starting days of their professional careers. 

The simplex method has also achieved great success in practice. As it is known, 
its applications to many fields, such as economy, commerce, production, science and 
technology, and defense and military affairs, etc, have brought about astonishing 
economic and social benefits. It is recognized as one of The Ten Algorithms in the 
Twenty Century IEEE2002; see Cipra (2000). 

After more than 70 years since its birth, LP is now a relatively mature but still 
developing branch. Nevertheless, there exist great challenges. The importance of 
large-scale sparse LP models is nowadays enhanced further by globalization. The 
everyday practice calls upon the research community to provide more powerful 
solution tools just to keep up with the ever-increasing problem sizes. Therefore, this 
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book does not only present fundamental materials but also attempts to reflect the 
state of the art of LP. It has been my long-lasting belief that research, in operations 
research/management science, in particular, should be of practical value, at least 
potentially. The author, therefore, focuses on theories, methods, and implementation 
techniques that are closely related to LP computation. 

This book consists of two parts. Part I mainly covers fundamental and conven- 
tional materials, such as the geometry of the feasible region, the simplex method, 
duality principle, and dual simplex method, implementation of the simplex method, 
sensitivity analysis, parametric LP, variants of the simplex method, decomposition 
method, and interior-point method. In addition, integer linear programming (ILP), 
differing from LP in nature, is also discussed not only because ILP models can be 
handled by solving a sequence of LP models but also because they are so rich in 
practice and form a major application area of LP computations. 

Part II presents published or unpublished new results achieved by the author 
himself, such as pivot rule, dual pivot rule, simplex phase-1 method, dual simplex 
phase-I method, reduced simplex method, generalized reduced simplex method, 
deficient-basis method, dual deficient-basis method, face method, dual face method, 
and pivotal interior-point method. The last chapter contains miscellaneous topics, 
such as some special forms of the LP problem, approaches to intercepting for primal 
and dual optimal sets, practical pricing schemes, relaxation principle, local duality, 
decomposition principle, and ILP method based on the generalized reduced simplex 
framework. 

To make the material easier to follow and understand, the algorithms in this book 
were formulated and illustrative examples were worked out wherever possible. If 
the book is used as a textbook for upper-level undergraduate/graduate course, (Part 
I) may be suitable to be as basic course material. 
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Since its birth, linear programming (LP) has achieved great success in many fields, 
such as economy, commerce, production, science, and technology, and brought 
about amazing economic and social benefits. After 70 years of development, LP 
now becomes a relatively mature branch in OR/MS. Nevertheless, the academic 
community faces a major challenge of the growing demand, which needs more 
powerful and robust tools to deal with large-scale and difficult problems. 

To meet the challenge, this book draws materials from a practical point of 
view, focusing on theories, methods, and implementation techniques that are 
closely related to LP computation. Its first edition consists of two parts. Roughly 
speaking, Part I covers fundamental materials of LP, and Part II includes pub- 
lished/unpublished new results achieved by the author. 

Little attention was received via the author’s ResearchGate Web page until Shi, 
Zhang, and Zhu published their book review in the European Journal of Operational 
Research (June 2018). Since then, the book quickly attracted considerable attention 
from academic communities around the world. As of November 6, 2022, the top ten 
chapter reads are as follows: 


Chapter Reads 
1. Duality Principle and Dual Simplex Method (Part I) 15419 
2. Simplex Feasible-Pointx Method (Part ID) 7861 
3. Integer Linear Programming (ILP) (Part I) 7340 
4. Implementation of the Simplex Method (Part I) 3284 
5. Dual Simplex Phase-I Method (Part IT) 2613 
6. Simplex Method (Part I) 2576 
7. Variants of the Simplex Method (Part I) 2328 
8. Geometry of the Feasible Region (Part I) 1951 
9. Pivot Rule (Part ID) 1844 


10. Simplex Phase-I Method (Part I) 1506 
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The preceding data are consistent with corresponding downloads from the 
Springer website. 

Surprisingly enough, the chapter Dual Principle and Dual Simplex Method” not 
only received the maximum number of reads as a whole but also almost weekly, 
even if it is devoted to such a classical topic. This might be because the discussions 
are comprehensive, compared with popular literature. 

Proposed for the first time, the Feasible-Point Simplex Method, which appeared 
as the final section of the chapter ’Pivotal Interior-Point Method,” received the 
second-highest reads as a whole. It achieved this in a short period of 10 weeks. 
Indeed, supported by the solid computational results, the method itself might exceed 
all expectations. 

The final chapter of Part I, ’Integer Linear Programming (ILP),” usually received 
the second-highest reads weekly. This happened, I guess, because many researchers 
are interested in the newly proposed controlled-branch method and controlled- 
cutting method, which have potential applications for developing new ILP solvers. 

The author’s main contributions seem to have not been fully valued by now, 
including most pivot rules, reduced and D-reduced simplex methods, deficient-basis 
and dual deficient-basis methods, face and dual face methods, and the decomposi- 
tion principle, included in Part II. This is somewhat surprising since these methods 
are supported by solid computational results, except for the newly introduced 
reduced and D-reduced simplex methods, and the decomposition principle. In the 
author’s view, the reduced simplex method is particularly noteworthy. It is the 
first simplex method that searches along the objective-edge. In contrast to existing 
simplex methods, it first determines pivot row and then pivot column, and does 
so without any selection essentially. As a dual version of it, the D-reduced simplex 
method shares similar attractive features, through determining pivot column first and 
pivot row later. As for the decomposition principle, it allows for solving arbitrarily 
large-scale (even dense) LP problems, in essence, giving a glimmer of light on some 
other types of separable large-scale problems. Time will likely clarify the value of 
these methods vs other LP solvers. 

We are also optimistic about the face and dual face methods. Without exploiting 
sparsity, the original methods were implemented using orthogonal transformation 
with favorable computational results reported. In the second edition, we added 
two chapters devoted to new methods with LU factorization for sparse computing. 
Indeed, this is a very natural development which the author cannot help himself 
from breaking into. 

Some changes on chapters and sections were made with corrections and improve- 
ments, and the whole second edition was organized into two parts. Roughly speak- 
ing, Part I (Foundations) contains Part I of the first edition, and Part II (Advances) 
includes Part II. The Simplex Feasible-Point Algorithm was improved, and removed 
from the chapter ’Pivotal Interior-Point Method” to form an independent chapter 
with its new title “Simplex Interior-Point Method,” since it actually represents a 
new class of interior-point algorithms, which can be transformed from the traditional 
simplex algorithms. The title of the original chapter was changed to “Facial Interior- 
Point Method,” since the remaining algorithms represent another new class of 
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interior-point algorithms, which can be transformed from the normal interior-point 
algorithms. In particular, the chapter “Integer LP” was rewritten in great gains of 
the introduction of the objective cutting. Another exciting improvement was the 
reduced simplex method. The original derivation of its prototype was presented in 
a chapter with the same title, and then transformed into the so-called “improved” 
one in another chapter. Fortunately, inspired by the bisection simplex method, we 
recently found a quite concise new derivation, so we can introduce it in a single 
chapter now. 

Finally, I am grateful to many individuals for their valuable comments and 
suggestions on the first edition of this book. In particular, I appreciate comments 
and encouragement given by Dr. Y.-Y Shi, Professor L.-H Zhang, and Professor 
W.-X Zhu. 


Nanjing, China Ping-Qi Pan 
November 2022 
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About the Book 


This book represents a real breakthrough in the field of linear programming (LP). 
Being both thoughtful and informative, it focuses on reflecting and promoting the 
state of the art by highlighting new achievements in LP. This new edition was orga- 
nized into two parts. The first part addresses foundations of LP, including geometry 
of feasible region, simplex method, implementation of simplex method, duality 
and dual simplex method, primal-dual simplex method, sensitivity analysis and 
parametric LP, generalized simplex method, decomposition method, interior-point 
method, and integer LP method. The second part mainly introduces contributions 
of the author himself, such as efficient primal/dual pivot rules, primal/dual Phase- 
I methods, reduced/D-reduced simplex methods, generalized reduced simplex 
method, primal/dual deficient-basis methods, primal/dual face methods, and new 
decomposition principle. 

Many important improvements were made in this edition. The first part includes 
new results, such as mixed two-phase simplex algorithm, dual elimination, fresh 
pricing scheme for reduced cost, and bilevel LP model and intercepting of optimal 
solution set. In particular, the chapter ’Integer LP Method” was rewritten in 
great gains of the objective cutting for new ILP solvers controlled-cutting/branch 
methods, as well as an attractive implementation of the controlled-branch method. In 
the second part, the “simplex feasible-point algorithm” was rewritten and removed 
from the chapter “Pivotal Interior-Point Method” to form an independent chapter 
with the new title “Simplex Interior-Point Method,” as it represents a class of new 
interior-point algorithms transformed from traditional simplex algorithms. The title 
of the original chapter was then changed to “Facial Interior-Point Method,” as 
the remaining algorithms represent another class of new interior-point algorithms 
transformed from normal interior-point algorithms. Without exploiting sparsity, the 
original primal/dual face methods were implemented using Cholesky factorization. 
To cope sparse computation, two new chapters for ones with LU factorization were 
added to the second part. Another exciting improvement was the reduced simplex 
method. In the first edition, the derivation of its prototype was presented in a chapter 
with the same title, and then converted into the so-called “improved” one in another 
chapter. Fortunately, the author recently found a quite concise new derivation, so 
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he can now introduce the distinctive and very promising new simplex method in a 
single chapter. 

This book is a rare work in LP. With a focus on computation, it contains many 
novel ideas and methods, supported by sufficient numerical results. Being clear 
and succinct, its content deduces in a fresh manner, from simple to profound. 
In particular, a larger number of examples were worked out to demonstrate 
algorithms. So, it is an indispensable tool for undergraduate/graduate students, 
teachers, practitioners, and researchers in LP and related fields. 


About the Author 


Ping-Qi Pan is a professor and doctoral supervisor of the Department of Math- 
ematics at Southeast University, Nanjing, China. He was a visiting scholar at the 
University of Washington (1986-1987) and visiting scientist at Cornell University 
(1987-1988). His research interest focuses on mathematical programming and 
operations research, especially large-scale linear optimization. He was standing 
council member of the Mathematical Programming Society of China and standing 
council member of the Operation Research Society of China. Professor Pan has 
received the honorary title of Outstanding Scientific-Technical Worker of Jiangsu 
Province of China. He was nominated as Top 100 scientist of 2012 by the Interna- 
tional Biographical Centre, Cambridge, England. He won the Lifetime Achievement 
Award by Who’s Who in the World in 2017. 


Notation 


In this book, in general, uppercase English letters are used to denote matrices, 
lowercase English letters used to denote vectors, and lowercase Greek letters do 
denote reals. Set is designated by uppercase English or Greek letters. Unless 
indicated otherwise, all vectors are column ones. 


LP 


Linear programming 

Integer linear programming 

Euclidean space of dimension n 

Euclidean space of dimension 1 

Origin of 7” (or the zero vector) 

Unit vector with the ith component | (dimension will be clear from the 
context; the same below) 

Vector of all ones 

Unit matrix 

Coefficient matrix of the linear program; also stands for index set of A’s 
columns, i.e., A = {1,...,} 

Number of rows of A 

Number of columns of A 

Right-hand side of the equality constraint 

Cost vector, i.e., coefficient vector of the objective function 

Basis (matrix). Also the set of basic indices 

Nonbasic columns of A. Also the set of nonbasic indices 

The jth column of A 

The entry of the ith row and jth column of A 

The jth component of vector v 

Euclidean norm of vector v 

The largest component of vector v 

Submatrix consisting of columns corresponding to index set J 
Subvector consisting of components corresponding to row index set [ 
Transpose of A 

Diagonal matrix whose diagonal entries are components of vector x 
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Vf (x) 
Ic? 
MUL 
IAT 
m\r 

p 

| 


« 
> 
O(@) 


ITI 

sign (t) 
range H 
null H 
dim T° 
cr 

int P 
La 


Tq) 


Notation 


Gradient of function f(x) 

IT is a subset of T 

Union set, i1.e., {t | tT € Mort ET} 

Intersection set, i.e., {t | 7 E€ H,t eT} 

Complementary set, i.e., {t | 7 ¢ M,t ¢T} 

Empty set 

Such that. For example, {x | Ax = b} means the set of all x such that 
Ax = b holds 

Far less than 

Far greater than 

Implies a number less than and equal to ka, where k is a fixed integer 
constant independent of a 

Absolute value of t if t is a real, or cardinality of t if t is a set 
Sign of real ¢ 

Column space of matrix H 

Null space of matrix H 

Dimension of set 

Number of combinations of taking m from n elements 

Interior of set P 

The largest integer no more than a 

The smallest integer no less than a 


This book involves the following two simplex FORTRAN codes: 


MINOS 


RSA 


The sophisticated smooth optimization package developed by Murtagh 
and Saunders (1998) at Department of Management Science and Engi- 
neering of Stanford University. Based on the simplex method, the LP 
option of this sparsity-exploiting code was used as the benchmark and 
platform for empirical evaluation of new LP methods 

The author’s private code based on the revised two-phase simplex 
algorithm without exploiting sparsity. It uses the Harris two-pass row 
rule 


Part I 
Foundations 


Chapter 1 ® 
Introduction Ghost for 


As intelligent creatures, human beings plan and carry out activities with pre-set 
objectives. Early human ancestors relied on their experience only, whereas their 
modern-day descendants make their decisions based on scientific tools. 

The word optimization means to achieve the best results with minimum effort. 
For example, how would one allocate limited resources (labor, money, materials, 
etc.) to maximize returns or minimize costs while meeting a specific demand. 
People collect information and data and express real-life problems with appropriate 
forms mathematically using numbers, functions, and equations, etc., through a so- 
called mathematical modeling process. Then, optimization methods/software are 
applied to deal with these models, consequently offering quantitative bases for 
sound decision making. This process has been revolutionized by the advent of the 
electronic computer. 

The linear programming (LP) branch emerged when the American mathemati- 
cian George B. Dantzig founded the LP model and the simplex method in 1947 (for 
the birth of LP, see Appendix A). The LP model has a simple structure, involving 
linear functions and equalities/inequalities; somewhat surprisingly, it turns out to be 
one of the most important and widely used models in the world. This seems to be in 
response to the words of Lao Tzu:! 


The Great Road Is the Simplest! 

Of course, “simplest” does not necessarily mean “easy.” The challenge is the huge 
scale of LP models that are difficult to handle. After 70 years of development, 
LP methods/software are now capable to solve models with considerably large 
scales. Even 30 years ago, for example, an LP model with 12,753,313 variables was 
solved successfully to provide an optimal scheme for an airline crew scheduling 
application (Bixby et al. 1992). Nevertheless, the academic community still faces 
major challenges to deal with large-scale and difficult models. Therefore, it might 


' Lao Tzu (about 571-471 BC) was an ancient Chinese philosopher. 
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be time to review the foundations as well as advances in LP and to provide a book 
for undergraduate/graduate students, teachers, researchers, and practitioners in LP 
and the related fields. 

In this chapter, we first introduce primitive notation and concepts of numerical 
algebra that are closely related to LP, including errors of floating-point arithmetic, 
the mathematical model of LP, illustrative applications, the standard LP problem, 
the basis matrix, and the basic solution. 


1.1 Error of Floating-Point Arithmetic 


As a basic tool for modern numerical computation, a computer is indispensable. 
LP without it is not imaginable. In using a computer, however, numerical errors are 
unavoidable because it can only represent a proper subset F of real numbers, and 
as a result, data can only be stored and calculated with finite digits. On one hand, 
primary data may be out of F and expressed approximately; on the other hand, 
numerical results yielded from subsequent operations of arithmetic may have to be 
rounded. Errors arising therefrom are called rounding errors. 

The set F of the floating-point values represented by a computer is characterized 
by 4 positive integers: the base B, precision (mantissa digits) t, and exponent range 
[L, U] (Forsythe et al. 1977, pp 10-29). More precisely, a computer can represent 
all nonzero numbers of the following form: 


fl(x) = £.a1...a; BY, O0<aj < fp, i=1,...,t, a, 40; L<e<U, 
(1.1) 


which together with 0 constitute F, and where a ...a; is called mantissa part and 


B® is the exponent part. 
We use the notation 


i=p M=p"(1- 6°). 


It is clear that for any floating-point number f(x) € F, it holds that m < |x| < M. 
Thus, F is a set of non-uniformly distributed rational numbers. Define the set 


S={xeER|m < |x| < M}. 
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Then, for any 0 # x e€ S, the approximate number f/(x), yielded from the 
computer, only holds mantissa of ¢ digits, via the following two methods: 


(i) 


(ii) 


Rounding method: It is based on the (¢ + 1)th digit of the mantissa of x. The 
digits at this position and after are all taken off if this digit is less than a half of 
B; in the other case, the digits are taken off after a unit is added to the previous 
digit. The associated absolute error bound is then 


1 e-t 
ee ed i (1.2) 
Truncation method: The (¢ + 1)th figure and after are all taken off. Thereby, 


Ffl(x) is the floating-point number nearest to x in F, whose absolute values are 
no more than |x|, and hence, the associated absolute error bound is 


| fix) — x] < pF. (1.3) 
The ¢ is a key quantity to numerical computations. To see this, define 
5 oS rounding 
€= pit truncating (1.4) 
Then, from either (1.2) or (1.3) it follows that 


| fix) — x| < «pe. 


Since 
|x| > .a,B° > 0, 
it holds that 
a = _ = €/a <€, 
that is 
fi@) =x +4), aise 


Therefore, € is just a relative error bound of the approximate number //(x). The 
so-called machine precision, €, is the smallest positive value in F, satisfying 


fld+0\ 41. 


Thereby, one may get to know the precision of a computer. 


6 1 Introduction 


Assume a, b € F, and represent any of the four arithmetic operations (+, —, *, /) 
by LJ. It is clear that if aLlb ¢ S, then the computer cannot handle and display 
overflow (when |aLIb| > M) or underflow (when 0 < |aLlb| < m) and terminate the 
computation (some computers and compilers place value 0 in the case of underflow). 
In normal cases, for the computed value f/(aLlb) of aLb, it is clear that 


fl(ab) = (ab)(1+8), 8] <€. 


Therefore, the relative error bound for a single arithmetic operation is €, a small 
number. Unfortunately, the situation may be entirely contrary when executing an 
algorithm or a series of operations. 

Usually, the algorithm’s performance may be affected by rounding or truncation 
errors greatly. Due to error accumulations, final results could be entirely meaning- 
less, as they could be far from the theoretical value. It is therefore important to take 
care of numerical errors in computations. In this aspect, the following simple rules 
should be followed: 


1. Avoid using a number close to 0 as the denominator or divisor. 

2. Avoid subtracting a number from another that is close to it. 

3. Avoid operating numbers with a very different order of magnitude, so that “large 
swallows small whole.” 

4. Avoid results close to m or M. 

5. Reduce the number of operations, especially multiplications or divisions. 


When 6 < 1, for instance, calculating from the right-hand side of each equation 
below gives a more accurate result than from the left-hand side. 


Vx +6 —JSx = 5/(V¥x 45+ 7x), 
—2 sin(x + 5/2) sin(6/2). 


cos(x + 6) — cosx 


Due to different errors introduced in computations, algorithms, which are 
equivalent theoretically, are often not equivalent numerically. An algorithm is said 
to be of good (numerical) stability if the resulting numerical error is under control 
and has little impact on the results. Unstable algorithms are unreliable. 

Usually, it is not an easy task to accurately evaluate the error bound of results 
computed by algorithms. Instead, the so-called backward error analysis is applied. 
Interested readers are referred to Wilkinson (1971). 


1.2 From Real-Life Issue to LP Model 


In practice, there are usually many schemes to be chosen by a decision maker, and 
different schemes may lead to widely diverging outcomes, as is crucial in fierce 
competition. This situation provides a wide stage for the decision maker to play. 
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Making decisions just based on experience cannot be mentioned in the same breath 
with mathematical tools. 

The formation of a mathematical model is the first step toward the application 
of LP. Based on full and reliable data gathered, a good model comes from one’s 
knowledge, skill, experience, and wisdom. This topic is beyond the scope of this 
book, and this section only gives the reader a taste of simple examples. 


Example 1.2.1 A manufacturer, who has 1100 tons of log, is committed by contract 
to supply 470 tons of lath to a company. There is a 6% loss in log when it is 
processed into laths. In addition, the production costs have risen since the signing 
of the contract, while the selling price of lath remains the same. In fact, he can 
make more profit by selling the log as raw material to another firm. What can the 
manufacturer do to have the maximum profit while still honoring the contract? 


This problem can be solved algebraically. Let x be the number of tons of logs the 
manufacturer can sell as raw material, and let y be the number of logs available to 
produce lath. It is clear that 


x+y = 1100, 


where x should be maximized to gain the maximum profit. On the other hand, since 
6% out of y is lost during the manufacturing process, the actual amount of lath 
produced is 


y — 0.06y, 
which, associated with the contract, must be equal to 470 tons, i.e., 
y — 0.06y = 470. 
Consequently, we have the following system of linear equations: 


Reece alee (15) 


1100-—y= x, 
which has a unique solution 
x = 600, y = 500. 


Therefore, the manufacturer should sell 600 tons of logs as raw material and use the 
rest 500 tons for the production of lath. 

In the preceding, the manufacturer only has a unique scheme to choose. Usually, 
decision makers face multiple or even unlimited choices, as shown in the following 
example. 
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Example 1.2.2 A manufacturer produces lath and sheet piles. Profit per lath is 
2 dollars, and profit per sheet pile is 5 dollars. The equipment capability of the 
manufacturer can produce at most 6000 lath or 4000 sheet piles per day. According 
to the sales contract, he can sell 3000 sheep piles at most, and the sales volume of 
laths and sheet piles cannot exceed 5000 per day. How should he arrange to produce 
lath and sheep piles to gain a maximum profit? 


Let x be the amount of lath, and let y be that of sheep piles to be produced per day 
(in thousand as the unit). For values of each pair of variables (x, y) corresponding 
to a decision, the associated profit is f(x, y) = 2x + 5y thousand dollars. So, what 
the manufacturer wants to do is to maximize the function value of f(x, y). 

It is clear that the yielded profit would be infinitely large if there was no 
restriction on the production. It is not the case, of course. According to the 
equipment capability, the average rate of producing lath and sheet piles is 6 thousand 
and 4 thousand per day, respectively. Therefore, variables x and y should satisfy 
inequality x/6 + y/4 < 1, or equivalently, 


2x + 3y < 12. 
According to the selling limitation, in addition, there is another inequality 
x+y Ss, 
and finally, 


y<3. 
In addition, the amount of output should be nonnegative. As the amount is large, we 
use the restriction x, y > 0 instead for simplicity and finally obtain integer results 
by rounding. 

The preceding analysis can be put into the following so-called mathematical 
model: 


max f(x,y) =2x+5y 
s.t. —2x —3y > —-12 


—-x-y>-5 (1.6) 
es a 
x, y=0, 


where x and y are called decision variables, and f (x, y) is called objective function; 
the inequalities are called constraints, (x, y) satisfying constraints is a feasible 
solution, and the set of all feasible solutions is the feasible region. Solving a model 
is trying to find its optimal solution, that is, a feasible solution maximizing the 
objective function over the feasible region. 
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Fig. 1.1 The feasible region y4 
of Example 1.2.2 


RY 


In contrast with Example 1.2.1, which has a single feasible solution, Exam- 
ple 1.2.2 involves infinitely many feasible solutions. In fact, the problem has the 
following feasible region: 


P={(x,y)|2x+3y< 12, x+y<5, y<3,x2=0,y =O}. 


Since there is a 1-to-1 correspondence between pairs of real numbers and points 
in a rectangular coordinate system, the feasible region P can be represented 
geometrically by Fig. 1.1, where x (thousand) represents the number of laths and 
y (thousand) that of sheep piles, and where each boundary is labeled by an equation 
corresponding to the closed half-plane. The feasible region P is just the intersection 
of these half-planes, within which each point corresponds to a feasible solution. 

Therefore, the problem now is how to find a point in P corresponding to the 
maximum possible value of function 2x + Sy. Since there are infinitely many points 
in P, itis impossible to pick up and compare them one by one, not to mention much 
more challenging large-scale problems with hundreds of thousands of variables and 
constraints. Although this simple instance can be solved by a graphical approach 
(Sect. 2.6), dealing with general LP problems requires sophisticate mathematical 
tools. 


1.3 Illustrative Applications 


LP has a very wide range of applications, touching upon almost all fields related 
to decision making and management. This section only brings up a few illustrative 
instances. Strictly speaking, some of these instances involve variables of nonneg- 
ative integer values and hence belong to the so-called integer/mixed-integer LP 
models, although the latter requirement is ignored here for simplicity. 


10 1 Introduction 


Example 1.3.1 (Production Planning) A factory produces furniture A, B, C. Each 
furniture production goes through 3 procedures: component processing, electroplat- 
ing, and assembling. The production capacity of each procedure per day is converted 
into effective working hours. Below are the required effective working hours and 
getatable profit for each piece of the furniture. 


Procedure Hours per piece Available hours per day 
Product A Product B Product C 
Component processing 0.025 0.05 0.3 400 
Electrofacing 0.20 0.05 0.1 900 
Assembling 0.04 0.02 0.20 600 
Profit (dollars/piece) 1.25 1.5 2.25 


Now how can the factory achieve the highest profit? 
Answer Let x1, x2, and x3 be yields of furniture A,B,C, respectively, and let f 
be the total profit. To gain the highest profit, construct the following model: 
max f = 1.25x, + 1.5x2 + 2.25x3 
s.t. 0.025x; + 0.05x2 + 0.3x3 < 400 
0.20x; + 0.05x2 + 0.1x3 < 900 
0.04x; + 0.02x2 + 0.20x3 < 600 


X1, x2, x3 20. 
An optimal solution obtained by the simplex method is 
x1 = 2860, x2 = 6570, x3 = 0, 


and the associated objective function value is f = 13430. That is to say, the factory 
should produce 2860 pieces of product A, 6570 pieces of product B, and no product 
C, achieving the highest profit 13430 dollars. 


Example 1.3.2 (Transportation) A company has 8800, 7200, and 5700 containers 
at ports A, B, C, respectively. These containers should be transported to plants 1, 
2, 3, 4, 5, whose working ability is 3200, 5300, 4100, 6200, and 2900 containers, 
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respectively. The following table lists the freight rate (dollars/container) of transport 
service from the ports to the plants: 


How should the company arrange to achieve the least total freight charges? 
Answer This is a balanced transportation problem, as the total number of 
containers is equal to the total number of containers, i.e., 


8800 + 7200 + 5700 = 3200 + 5300 + 4100 + 6200 + 2900 = 21700. 


Let xij, i = 1,2,3; 7 = 1,...,5, be the number of containers that are 
transported from ports A, B, C to plants 1-5, and let f be the total freight charges. 
The model is as follows: 

min f = 55x11 + 150x12 + 78x13 + 38x14 + 90x15 
+25x21 + 60x22 + 25x23 + 43x24 + 35x25 
+102x3; + 80x32 + 52x33 + 74x34 + 60x35 
s.t. Xqp + x12 +413 + x14 + x15 = 8800 
X21 + x22 + x23 + x214 + x25 = 7200 
X31 + x32 + x33 + x34 + 135 = 5700 
X11 + x21 + x31 = 3200 
x12 + x22 + x32 = 5300 
X13 + x23 + x33 = 4100 
X14 + X24 + x34 = 6200 
X15 + x25 + x35 = 2900 
xij 20, i=1,2,3; j=1,...,5. 


An optimal solution obtained by the simplex method is 
X11 = 2600, x14 = 6200, x2; = 600, x23 = 4100, x25 = 2500, x32 = 5300, x35 = 400, 


and all the other variables take value 0. Then the company pays the minimum total 
freight 1031600 dollars. 
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Example 1.3.3 (Burdening) To feed animals, someone needs 30 grams of mineral, 
700 grams of protein, and 100 milligrams of vitamin per day, at least. There are 
5 kinds of feed to buy. The nutrition and price per kilogram for each feed are as 
follows: 


Fodder | Mineral (g.) | Protein (g.) | Vitamin (mg.)) | Price (dollars 1/kg.) 


A 0.5 18.0 0.8 0.8 
B 2.0 6.0 2.0 0.3 
Cc 0.2 1.0 0.2 0.4 
D 0.5 2.0 1.0 0.7 
E 1.0 3.0 0.5 0.2 


Find a buying scheme to have a minimum cost to serve animals’ needs. 

Answer Let x;(kg.), j = 1,...,5, be the purchase quantities of the 5 kinds of 
feed, respectively, and let f be the total cost. The goal is to determine the values of 
these decision variables to achieve the minimum cost. The model is as follows: 

min f =0.8x; + 0.3x2 + 0.4x3 + 0.7x4 + 0.2x5 
s.t. 0.5x1 + 2.0x2 + 0.2x3 + 0.5x4 + x5 > 30 
18.0x; + 6.0x2 + x3 + 2.0x4 + 0.3x5 > 700 
0.8x, + 2.0x2 + 0.2x3 + x4 + 0.5x5 > 100 
xj =0, j=1,...,5. 


An optimal solution is obtained by the simplex method, i.e., 
xy = 25.6, x2 = 39.7, x3 = x4 = x5 = O, 


with minimum objective function value f = 32.28. So, he should buy 25.6 kg. of 
feed A, 39.7 kg. of feed B, and none of the other 3 feeds with the minimum cost 
32.28 dollars. 


Example 1.3.4 (Human Resource Arrangement) A round-the-clock supermarket 
arranges salespersons. Every day falls into 6 time intervals, 4h each. Every 
salesperson starts work at the beginning of the intervals and then continuously works 
for 8h. The required numbers of salesperson for each of the intervals are as follows: 


Interval 1 2 3 4 5 6 
2:00-6:00 | 6:00-10:00 | 10:00-14:00 | 14:00—-18:00 | 18:00-—22:00 | 22:00-—2:00 
Number i; 15 25 20 30 7 
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How to arrange to achieve the least number of salespersons per day? 


Answer Let x;, j = 1,...,6, be the number of salespersons who start work at 
the beginning of interval j, and let f be the total number of salespersons. The goal 
is to determine the variable values to achieve the minimum number of salespersons 
working per day. The model is as follows: 


min f =x +42 +%3 +%4 + x5 + X6 
S.t. x] +x6>7 
xX + x2 > 15 
x2 + x3 z25 
x3 4+Xx4 > 20 
X4 + x5 > 30 
x5+x%6>7 


xj >0, f=1,...,6. 


An optimal solution obtained by the simplex method is 


x2 = 25, x4 = 30, x6 = 7, X41 = x3 = x5 = 0, 


and the associated objective function value is f = 62. So, the business should 
arrange 25 salespersons working from 6:00, 30 ones working from 14:00, and 7 
from 22:00. This scheme requires only 62 salespersons. 


Example 1.3.5 (Laying-off) A shop is asked to make 450 steel frames. Each frame 
needs 1.5, 2.1, and 2.9 meters long angle steel. Each raw material is 7.4 meters long. 
Find the optimum laying-off way. 


Answer A direct way is to make a frame using raw material to get the 3 kinds 
of angle steel. If so, there would be a need for 450 raw materials, each of which 
would leave an 0.9 meters long offcut. To obtain the optimum scheme, consider the 
following ways, each of which would leave an offcut no more than 0.9 meters long. 


Length (meters) Way 
A B C D E F 
2.9 1 2 O 1 0 1 
2.1 O° Oe 2 «2 1 1 
1.5 3 1 2 0 3 1 
Total 74 = 73 7.2 7.1 66 6.5 
Offcut 0.0 0.1 0.2 03 08 0.9 
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These ways should be mixed and matched to gain the minimum total length of 
offcuts. Let x;, j = 1,...,6, be numbers of raw materials taken, corresponding 
to ways A, B,C, D, E, F, and let f be the total length of offcuts. Construct the 
following model: 
min f =0.1x2 + 0.2x3 + 0.3x4 + 0.8x5 + 0.9x6 
s.t. xy + 2x2 + x4 + x6 = 450 
2x3 + 2x4 + x5 + x6 = 450 
3x, + x2 + 2x3 + 3x5 + x6 = 450 
xj =0, j=1,...,6. 


The optimal solution given by the simplex method is 
x, = 135, x2 = 45, x4 = 225, x3= x5 =x = 0, 


with objective function value f = 72. Therefore, the shop should use 135 raw 
materials for way A, 45 for way B, and 225 for way D, with a total length of 72 
meters of offcuts. The minimum total number of used raw materials is 405. 

The sizes of the preceding problems are very small, which are just for the reader 
to get a taste of LP applications. Nowadays, real-world problems are usually very 
large. Even so, small problems still help to convey ideas and illustrate algorithms. 


1.4 Standard LP Problem 


Some of LP problems in the previous section seek for maximizing the objective 
function, while others seek for minimizing it. Besides, some of their constraints 
are equalities, while others are inequalities of “>” or “<.’ For convenience, we 
introduce the following problem in a so-called standard form: 


min f = cjx] + ¢2x2 +--+ + CnXn 


S.t. Q41X1 + aj2xX2 + +++ + AjynXy =b, 
azix1 + az0x2 +++++4nXn =b2 

; (1.7) 
Am1X1 + Am2xX1 +++++amnXn = bm 


xj 20, j=Hl,...,n. 
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All kinds of LP problems can be transformed into standard forms: 


1. Any maximization problem can be transformed into a minimization problem by 
introducing the negative objective function instead. 
Using f = —f’, e.g., the objective 


max f’ = cyx1 + cx. +-++ + CnXp 


can be replaced by 


min f = —cyx1 — c2xX2 — +++ — CnXp. 
It is clear that this does not matter to the optimal solution but changes the sign of 
the optimal objective value only. 
2. Any inequality can be transformed into equality by introducing an extra nonneg- 
ative variable. 
Introducing a so-called slack variable x41 > 0, e.g., the “<” type of inequality 
ax] +a2x2 +--+ + axe <= B 
can be transformed into equality 
OX] + 2x2 +--+ + ORxE + XK41 = B, 
and the “>” type of inequality 
yx, +a2x2 +--+ t+ apx, = B 
can be transformed into the equality 


OX] + ax. + +--+ ORXE — XK41 = B. 


3. Nonnegative constraints on variables can be obtained by introducing auxiliary 
variables. 


Using variable transformation Xp = —Xxk, €.g., nonpositive constraint Xp < 0, can be 
converted into x; > 0. A free variable, say x, can be eliminated by the following 
transformation: 


/ 
Xp = Xk — Xk+1, Xk, Xk+1 = O. 
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Example 1.4.1 Transform the following LP problem into the standard form: 


max f’ = —2x1 + x2 — 5x; 
s.t. —3x( +s +eq = 2 
X1 = 1 ae, >-3 
ay = dag = 2x, = I 
x1,x2 > 0. 
Answer Maximizing f’ is transformed into minimizing f = —f’. The free 


variable x4 is substituted by x; = x3 — x4, x3,x4 > 0. The first 3 inequalities are 
transformed into equalities by introducing slack variables x5, x6, x7, respectively. 
The resulting standard problem is as follows: 


min f = 2x; — x2 + 5x3 — 5x4 


S.t. —3x, + xo+ 43- x4 t+ x5 = 2 
Xx, — 7x2 + 4x3 -— 4x4 — X6 =-3 
Xx, — 3x2 — 2x3 4+ 2x4 +x7= 1 


xj >0, j=l,...,7. 
Using vector and matrix notation 
A=Gij), x= Oy, .an)', C= (C1-..en)", b= (b1,.-- bm)", 
we put the standard problem (1.7) into the compact form below: 
min f =c'x (1.8) 
s.t. Ax =), x>0, , 
where A € R”*", ce RR", be R”, m <n. It is assumed that rank (A) = m, 
except indicated otherwise. 
It goes without saying that each row or column of coefficient matrix A does 
not vanish. f = c!x is called objective function, and c is price (cost). The system 
Ax = bis called equality constraint, and x > 0 nonnegative constraint. 


Usually, theories/methods are developed to deal with standard problems, 
although they are essentially applicable to general LP problems. 


Definition 1.4.1 A solution fulfilling the constraints is a feasible solution, and the 
set of all feasible solutions is the feasible region, 1.e., 


P={xeR"| Ax =b, x > 0}. (1.9) 


A feasible solution with the minimum objective function value over the feasible 
region is an optimal solution. 
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The minimization problem is called (lower) unbounded if its objective value in 
the feasible region can be arbitrarily small. 


1.5 Basis and Feasible Basic Solution 


The standard LP problem (1.8) involves the linear system of equations, Ax = b, 
as equality constraint. Note that it is a so-called underdetermined system, as the 
number of equations is less than that of unknowns. 

The set of all solutions is called solution set of the system. Systems are equivalent 
if they have the same solution set. There are two basic types of equivalent transfor- 
mations into systems. In this respect, the validity of the following propositions is 
evident. 


Proposition 1.5.1 A system, resulting from multiplying any equation of it by a 
nonzero, is equivalent to the original. 


Proposition 1.5.2 A system, resulting from adding a multiple of any equation to 
another, is equivalent to the original. 


Any of the preceding operations is called elementary (row) transformation. 
The second type of elementary transformation is especially important since it can 
eliminate a nonzero entry of A. Using a series of such kinds of transformations, e.g., 
the Gauss—Jordan elimination, converts a linear system into a so-called canonical 
form that is readily solvable. 

Let us bring up an example of 3 x 5 standard LP problem: 


min f =x; +2x2-—x4+ x5 


s.t. 2x1, + x2 + 3x3 + 2x4 =..5 
xy —xX2 +2x3-—x4+3x5 = 1 (1.10) 
x1 — 2x3 —2x5 =-1 


xj 20, fHl,...,5. 


For convenience, we strip the augmented coefficient matrix from the linear system 
and put it into the following tableau: 


1 -l 2 —1 3 1 
1 =? —2 =1 


where empty cells stand for value 0 (the same below). 
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The preceding tableau and the system itself are regarded as the same. Carrying 
out elementary transformations with the tableau amounts to doing the same thing 
with the system itself. 

A so-called pivot is a nonzero entry in the tableau, which determines a Gauss— 
Jordan elimination process. As an example, take the first entry 2 in the first column 
as pivot. Multiply by 1/2 the first row to turn the pivot to 1, leading to the following 
tableau: 


i 1d 372" 5/2 
i. “Sh 2° a, B 1 
1 =) =o | 1 


Then add —1 times of the first row to the second and third row, respectively, to 
eliminate entries in the first column except the pivot, giving the following tableau: 


xi x2 X3 X4 x5 RHS 
1 1/2 3/2 1 5/2 
—3/2 1/2 —2 3 —3/2 

1/2 7/2 1 2 7/2 


Take the second entry in the second column as the next pivot. Multiply by —2/3 
the second row to turn the pivot to 1, leading to the following tableau: 


xX] x2 X3 X4 xe RHS 
1 1/2 3/2 1 5/2 
1 -1/3 4/3 -2 1 


1926 PS, at - | Se 


Add —1/2 and 1/2 times of the second row to the first and third rows, 
respectively, to eliminate entries in the second column except the pivot, giving the 
following tableau: 


Xi = X2 x3 X4 x5 RHS 
1 5/3 1/3 1 2 
1 -1/3 4/3 —2 1 


Lia 1/3i 3 3 
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Take the third entry of the third column as pivot. Multiply by —3/11 the third 
row to turn the pivot to 1, leading to the following tableau: 


a) x3 X4 x5 RHS 
1 5/3 1/3 1 2 
1 -1/3 4/3 —2 1 


a Se 


Add —5/3 and 1/3 times of the third row to the first and second rows, 
respectively, to eliminate entries in the third column except the pivot, giving the 
following tableau: 


xX] x2 X3 X4 X5 | RHS 
1 2/1 4/11 | 7/11 
1 15/11 —19/11 | 14/11 


1 Wl oft | 9st 


Consequently, the preceding tableau includes a 3 x 3 unit matrix, and Gauss— 
Jordan elimination is completed. The corresponding so-called canonical system can 
be written as 


xy = 7/11 — (2/1) x4 + (4/1) xs 
xy = 14/11 — (15/11)x4+ (19/11) xs (1.11) 
x3 = 9/11 — (1/11)x4 — (9/1) xs, 


which is equivalent to the original system (Propositions 1.5.1 and 1.5.2). However, 
the preceding is so simple that itself exhibits general solutions of the system if x4 
and x5 are viewed as independent variables. 

In particular, setting zero to both x4 and x5 leads to 


mp=7/ll, x2 =14/11, x3=9/11, x4=2x5=0, 


which is said to be basic solution. In fact, this solution can be directly read from the 
RHS side of the last tableau. 

Nevertheless, Gauss—Jordan elimination does not always go smoothly if pivots 
are determined in the natural order. The process breaks down if some pivot happens 
to be zero. Even if all pivots are nonzero, the outcome may still be useless if some 
pivot is too close to zero (Sect. 1.1). 
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Table 1.5.1 Canonical x; lxe |--- lx. |x; se RHS 
ji ja | jr drt | Xin 
tableau i= = = 
1 Gin jn Gi, jn | Pin 
1 Gin jr Gir jn | Pin 
1 Gi, jrgs Gi, jn | Bi, 
Gist jt Ging in | Ping 
|? 
~ = = 
Gin jt Gin jn | Pim 


Let us turn to the so-called Gauss—Jordan elimination with complete pivoting. 
Put the augmented matrix (A|D) to the following tableau: 


xX] x2 Xn RHS 
M1 a2 +s Ain by 
a2, 422,— + An by 
Gm1 m2 joe ann __ Din 


Without loss of generality, suppose that the first r pivots are in the natural order 
and eliminations are performed such that the r x r northwest corner submatrix of A 
becomes a unit one, as shown in Tableau 1.5.1. 

At the next step, select a), from entries within the non-pivot rows and columns 
such that 

ldpq| = max{|G; j| | i =Urtis---sims J = Srtis +++ Jn}- (1.12) 
Assume that it is nonzero. Taking it as the (r + 1)-th pivot, we multiply row p by 
1/dpq and add —ajq, (i = ir+1,.-., im; i A p) times of the row to corresponding 
rows, respectively. To complete this step, we perform row and column permutations 
such that the (r + 1) x (r + 1) northwest submatrix becomes unit. 

The process terminates ifr = m, orr < m but dpqg = 0, that is, the (m —r) x 
(n — r) southeast submatrix vanishes. The final tableau is called canonical tableau. 
It is clear that if r = m, orr < m but bi,, Vt =r+41,...,m, vanish, and hence all 
non-pivot rows can be dropped from the tableau, system Ax = D is consistent. 

According to linear algebra, such a key positive integer r equals the rank of A. 
Thus, the following can be claimed. 


Theorem 1.5.1 The system Ax = b (m <n) has solutions if and only ifr = m, 
orr <™m but all entries in the non-pivot rows of the right-hand side vanish. In the 
latter case, it has infinitely many solutions. 
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The index set B = {j1,..., j,} of pivot columns is said basic, and N = A\B = 
{jr+1,--+> Jn} Said nonbasic. The associated variables are said basic and nonbasic 
correspondingly. 

Recall that for the standard LP problem, it is usually assumed that rank A = m. 
That is to say, Ax = b is always consistent, and the final canonical tableau offers a 
set of solutions to Ax = J, Le., 


xp =b—Nxy, 
XN =1XN; 


where the nonbasic variables of xy on the right-hand side are viewed as indepen- 
dent. 


Definition 1.5.1 The solution resulting from setting the nonbasic variables to zero 
is a basic solution. If its basic components are all nonnegative, it is a feasible basic 
solution. 


Thus, the canonical tableau corresponds to the basic solution 
Xp = b, xy =0, 


which is a feasible basic solution if xg > 0. 


Theorem 1.5.2 A feasible solution is a feasible basic solution if and only if the 
columns of A corresponding to its positive components are linearly independent. 


Proof Necessity. It is clear that all positive components of a feasible basic solution 
are basic ones and hence correspond to linearly independent columns. 

Sufficiency. Assume that a feasible solution has s positive components corre- 
sponding to linearly independent columns. Since rank A = m, it holds that s < m. 
If s = m, then the feasible solution is clearly a basic solution. If, otherwise, s < m, 
there exist additional m — s columns of A, which and the s columns together 
constitute a linearly independent set of columns. Clearly, the solution is just the 
basic one associated with the canonical form, corresponding to the set taken as basic 
columns. Oo 


Definition 1.5.2. A nonsingular m x m submatrix of A is a basis. It is a feasi- 
ble/optimal basis if the corresponding basic solution is feasible/optimal. 


There exists a basis for the standard LP problem since A has full row rank r = m. 
Therefore, Ax = b has an infinite number of solutions. The concept of basis plays 
a central role in the simplex methodology. 

By Gauss—Jordan elimination with column/row permutations, any basis can be 
converted into m x m unit matrix, and hence, (A|b) converted into a canonical 
tableau. It is to say, a basis corresponds to a canonical tableau, and hence a 
unique basic solution. But it is not a 1-to-1 correspondence: a basic solution could 
correspond to multiple bases, which is closely related to the so-called degeneracy 
(Sect. 3.4). 
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Based on the fact that the objective function is continuous over a closed feasible 
region, we conclude this section with the following claim: 


Proposition 1.5.3. The answer to any LP problem must be one of the following three 
cases: 


(i) Infeasible problem: There exists no feasible solution. 
(ii) Unbounded problem: There exists a feasible solution, but the feasible value is 
lower unbounded over the feasible region. 
(iii) There exists an optimal basic solution. 
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Chapter 2 ® 
Geometry of Feasible Region oes 


The feasible region P, defined by Definition 1.4.1, is of great importance to the LP 
problem. Theories and methods of LP are closely related to P, without exception. 
In this chapter, we introduce its special structure in geometry, including P as a 
polyhedral convex set, interior point, relative interior point, face, vertex, extreme 
direction, representation of P, optimal face and optimal vertex, graphic approach to 
LP, heuristic characteristic of an optimal solution, and feasible direction and active 
constraint. 

In general, P can be degenerate or even empty. The following lemma (Farkas 
1902) renders a sufficient and necessary condition for nonempty P (for the proof, 
see the end of Sect.5.2). Readers can refer to the literature on the basic concepts 
and operations of 7?” space. R” space. 


Lemma 2.0.3 (Farkas) Assume A € R'"*" and b € R". The feasible region P is 
nonempty if and only if 


b’y > 0, Vyef{yeR™ | Aly > 0}. 


Figure 2.1 serves as a geometric explanation for the preceding Lemma, where 
a1, a2, a3 are the columns of A € R2*3. 

The area Y between the vectors v and w corresponds to all vectors that form an 
angle no more than 2/2 with each column vector of A. It is seen that each y € Y 
forms an angle no more than 2/2 with vector bj, but not with b2. Therefore, P is 
nonempty when b = b, but empty when b = bp. 

Thereafter in this book, discussions will be based on the following: 


Assumption 1 The feasible region P is nonempty and infinite. 
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Fig. 2.1 A geometrical a3 a2 
explanation for Farkas’ w 
Lemma 


ay 


be 


2.1 Feasible Region as Polyhedral Convex Set 


For any given two points x, y € R”, the set 
S=f{ax+(1l—-a)y|aeR} 


is a (straight) line through them. If 0 < a < 1, it is an open segment with endpoints 
x and y, denoted by (x, y); if 0 < a < 1, itis a closed segment, denoted by [x, y]. 
Hereafter, a so-called segment will be a closed one, unless indicated otherwise. 


Definition 2.1.1 [I is an affine set if it includes the whole line through any two 
points whenever it includes them. The affine hull of a set is the smallest affine set 
that includes the set. 


Lines in R? and planes in R? are instances of affine sets. The whole space R” is 
an affine set. An empty set and a single point set are viewed as affine sets. It is clear 
that the intersection of several affine sets is an affine set. 

For any givena;, i = 1,...,k satisfying ea a; = 1, the point 


k 
x= So ax 
i=1 


is called an affine combination of x',...,x*. It is easy to show that the set of all 
such affine combinations, i.e., 


k k 
Yoox’ | lai =1, a ER, i hcad 
i=l 


i=1 


is an affine set, called affine hull of these points. The two points in Definition 2.1.1 
can be replaced by multiple points: it is easy to show that T] is an affine set if and 
only if the affine hull of any finitely many points within IT belongs to I. 

The set L is a subspace of R” if it is closed for all linear operations, that is, for 
any x,y € Landa, B € FR, it holds that ax + By € L. 
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Theorem 2.1.1 An affine set is an affine subspace if and only if it includes the 
origin. 


Proof The necessity is clear. Sufficiency. Let IT be an affine set including the origin. 
Then for any x € II anda € FR, it holds that 


ax=ax+(1—a)0eTl. 
On the other hand, it holds for any x, y € IT that 
(x + y)/2 = (1/2)x + A — 1/2)y € TI. 


Therefore, IT is closed for linear operations and is hence an affine subspace. oO 


Theorem 2.1.2 For any nonempty affine set Tl, there exists vector p so that 
L={x+p|x € Ty} 


is an affine subspace, and such subspace is unique. 


Proof According to Theorem 2.1.1, I] is an affine subspace if 0 € IT. Note that 
L = [1 corresponds to p = 0. Assume that 0 ¢ II. Since II ¥ 9, there exists 
OA y ell. Letting p = —y, it is clear that 


L={x+p|xeT} 
is an affine set including the origin and is hence an affine subspace. 


Now let us show the uniqueness. Assume that L), L2 are affine subspaces such 
that 


Li={y+pily € Wj, Ly ={y+ p2| y € UW}. 
It is clear that 
T= {x — pi |x € Li}. 
If p = —p1 + po, therefore, it holds that 
L2={x+ p|xe Ly}, 
from which it follows that x + p € L2 for any x € L;. Since 0 € Lj, p € Lz holds. 


Further, since L2 is a subspace, x = (x + p) — p € Lz holds. Therefore, £1 C Lo. 
L2 C L; can be similarly derived. So it can be asserted that L; = Lp. oO 


Geometrically, the affine subspace L along the vector p is parallel to the affine 
set IT. It is therefore called a parallel subspace of II. The dimension of L is said to 
be that of II, which is equal to the number of independent components (coordinates) 
of elements in IT. It is clear that an affine set is unbounded. 


26 2 Geometry of Feasible Region 


Let a be a nonzero vector, and let 7 be a real number. The set 
H={xeR"|a'x =n} (2.1) 


is called hyperplane, whose normal vector is a (a L A); in fact, for any two points 
x, y € H, it holds that 


al(x —y)=a'x —aly=n-—n=0. 


It is clear that any hyperplane is an affine set. 
Any straight line in R? and any plane in R? are instances of hyperplanes. 


Definition 2.1.2 The signed distance from point x to hyperplane H given by (2.1) 
is 


(a'% —n)/llal. 


If r = 0, the point x is within H. It might be well to assume 7 > 0. Then, if 
r <0, the origin and x are both on the same side of H; if r > 0, the two points are 
on different sides of H. 

The hyperplane associated with the objective function is of significance to LP. 
Regarding objective value f as a parameter, the sets 


H(f) = {x|clx = f} 


are a family of isosurfaces of the objective function. The gradient V f = c of the 
objective function is the common normal vector of all objective isosurfaces, pointing 
to the increasing side of the objective value. 

The following gives a general expression of affine set. 


Theorem 2.1.3 The set I is an affine set if and only if there exists W ¢ R‘*" and 
h € RE such that 


Tl = {x | Wx =A}. (2.2) 


Proof We might as well rule out the trivial case when II is empty or the whole 
space. 
Sufficiency. Let I] be defined by (2.2). For any x, y € II and real q, it holds that 


Wax+(1-—a)y)=aWx+(1-—aWy=ah+(1-ah=h, 


leading to ax + (1 — a)y € TI. Therefore, IT is an affine set. 
Necessity. Let IT be an affine set. Assume that L is a parallel affine subspace, and 
w!,..., w* are basis of the orthogonal complementary space of it. Then, it follows 


that 


L={y | (w')'y =0, i=1,...,4} = {y | Wy =0}, 
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where rows of W € R**" are (w!)!,..., (w*)". Introduce h = Wp. Since L is a 
parallel subspace of IT, there exists a vector p such that 


L={x-—p |x eT. 


Thus, for any x € I, it holds that x — p € L; and W(x — p) = 0 means that x € 
{x | Wx =h}. Ifx € {x | Wx = h}, conversely, then Wx — Wp = W(x — p) = 0; 
hence, from x — p € L, it follows that x € IT. So, I has expression (2.2). oO 


The preceding says that a set is an affine set if and only if it is the intersection of 
finitely many hyperplanes. It is clear that the dimension of the affine set is equal to 
n — rank(W). In particular, the solution set 


A={x | Ax=)} (2.3) 


of the constraint system of the standard LP problem is an affine set, in which 
rank A =m, A #@anddim A=n—~m. 


Definition 2.1.3 C is a convex set if it includes the whole segment whenever it 
includes its two endpoints. The smallest convex set including a set is the convex 
hull of the latter. 


Quadrants and disks in R? and octants and spheres in R? are instances for convex 
sets; segments and the whole space are also convex sets. An empty set and a single 
point set are regarded as convex sets; intersections of convex sets are convex. 

It is clear that any affine set is convex, but a convex set is not an affine set in 
general, e.g., disks and spheres are not affine sets. Note that any convex set has an 
affine hull. The dimension of the latter is said to be the dimension of the former. We 
might as well assume that the convex sets are all closed ones. 

For any aj > 0, i= 1,...,k satisfying a1 a; = 1, the point 


k 
x= ae 
i=1 


is called a convex combination of x!,..., x*. It is obvious that the set of all such 
convex combinations, i.e., 


k k 
yo! |) Yaa a= 0, fateh 
i=l i=l 


is the convex hull of these points. Any set C is a convex set if and only if C includes 
the convex hull of any finite number of points within C. 
Any hyperplane H divides the whole space into two closed half-spaces, i.e., 


Hy ={x|a™x <n}, and Hr={x|a™x >n}. (2.4) 
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The intersection of an infinite number of closed half-spaces is called polyhedral, and 
bounded polyhedral called polyhedron. It could degenerate to a segment or point, or 
even an empty set. 

It is clear that any polyhedral/polyhedron is a convex set. 

A convex set C is called polyhedral convex cone if ax € C holds for any x € C 
and a > 0. It is easy to show that a set is a polyhedral convex cone if and only if it 
is the intersection of finitely many closed half-spaces passing through the origin, as 
expressed by {x € R” | Ax > O}. 

The constraint x > 0 in the standard problem corresponds to the positive octant, 
which is the intersection of the n closed half-spaces with the coordinate planes as its 
boundary. Therefore, the feasible region P is the intersection of the affine set A and 
the positive octant. As any hyperplane a'x = n may be viewed as the intersection 
of two closed half-spaces (2.4), P may also be viewed as the intersection of a finite 
number of closed half-spaces. Therefore, we make the following statement. 


Proposition 2.1.1 The feasible region P is a polyhedral convex set. 


2.2 Interior Point and Relative Interior Point 


The feasible region itself could be degenerate, as it is closely related to the real 
dimension of it. This section is devoted to this topic. 
Introduce the following index set: 


J={j eA | xj =0, xe P}, (2.5) 


which is said to be the index set of zero components. It is noted that if x € P and 
xj > 0, then j ¢€ J. 

Proposition 2.2.1 Denote by I; the coefficient matrix of the systemx; =0, j € J’. 
Then it holds that 


n—min{m+|J|, n} < dim P =n —r <n —max{m, |J]}, (2.6) 


r = rank i 
Tj 


Proof Note that rank A = m and P is nonempty infinite, according to the basic 
assumption. 
It is clear that |J| <r and 


where 


P={xER"| Ax =b, x >0; xj =0, fe J). 
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IF |J| =r, thenx; =0, j € J is acanonical form of system 
Ax = b; xj =0, JéJ. 


If |J| <r, besides x; j € J, there are additional r — |J| basic variables, and hence 
a canonical form of the preceding system. It is to say that there exists a canonical 
form, whose n — r nonbasic variables do not belong to J. Therefore, it holds that 
dim P =n —r, and (2.6) follows from min{m + |J|, n} > r > max{m, |J|}. oO 


In particular, dim P =n —mif J = @. 

Usually, if x € P and x > 0, it is called an interior point. If J 4 @, therefore, 
the feasible region P has no interior point in the normal sense, as is often the case 
for real problems. For the sake of application, we introduce the following concept. 


Definition 2.2.1 Let A be defined by (2.3), and let J be defined by (2.5). Assume 
that J # % and x e€ P. If there exists 6 > O such that a neighborhood of x is 
included in P, i.e., 


Q(6) ={x EA| xj =0, f EJ; Ix — Z| <8} C P, (2.7) 


then x is a relative interior point of P; otherwise, it is a boundary point. 


The set of all relative interior points of P is called its relative interior, denoted 
by int P. 


Theorem 2.2.1 Assume x € P. Then x € int P if and only if 
xj>0, jeJ. (2.8) 
Proof Sufficiency. Let the point x € P satisfy (2.8). Using 


6 = min x; > 0 (2.9) 
jel 


and 
Q(5) = {xe Al xj =0, je J; lx —Xl] < 3}, 
then for any x € (Q2(6), it holds that 
xeA and xj=0, jeJ. 


Moreover, since 
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it holds that 
|x; -—xjl<|lx-xIl <5, jeJ, 


which together with (2.9) gives 


xj>dj;-5>0, jed. 


Thus x € P. Therefore (6) C P, and x is an interior point of P. 

Necessity. Assuming x ¢€ int P, then there is 5 > 0 such that 02(6) C P; also 
there is p € J such that x, = 0. According to the definition of J, there is x’ € P 
such that x, > 0. It is clear that for any a > 0 there exists 


x=-ax’+(l+a)¥ =xX+a(x—x Ed. (2.10) 
Hence, when a is sufficiently small, it holds that 
IIx — Xl] = az — x" <6. 


In addition, it is clear that xi, x; =0, j € J,andhence x; =0, j € J. Therefore, 
x € {2(6). On the other hand, from (2.10), x» = 0, anda > 0, x), > 0, it follows 
that 


Xp= =a +(14+@)Xp = =O, <0, 


which contradicts (6) C P. Therefore, (2.8) holds if x € int P. oO 
Proposition 2.2.2 /fdim P > 1, then P has a relative interior point. 


Proof The assumption of the proposition implies J 4 J, because otherwise it holds 
that J = A and hence dim P = 0, leading to contradiction. On the other hand, for 
any j € J, there exists x € P such that x; > 0; hence from the convexity of P, it 
follows that there is x € P satisfying x; > 0, j € J. According to Theorem 2.2.1, 
it is known that x € int P £9. Oo 


Example 2.2.1 Investigate the interior of the feasible region of the following 
problem: 


min xj +x2+%3+%x4+45, 

S.t. xy — x3+x44+ x5 = 6, 
x2 — x3—- x4 —x5 = 0, 
—x2 + 3x3 + x4 + x5 = 0, 
xj 20, j=l,...,5. 
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Answer Adding the second constraint equality to the third gives 
2x3 = 0. 


It is clear that the feasible region is nonempty, and the x3 component of all feasible 
points equals 0. Eliminating x3 from the problem leads to 


min xj + x2 +%4+ 4X5, 
S.t. x] +x4+ x5 = 6, 

x2 —x4—- x5 = 0, 
xj 20, fj =1,2,4,5, 


the interior of the feasible region of which is clearly nonempty, corresponding to the 
relative interior of the feasible region of the original problem. 


2.3 Face, Vertex, and Extreme Direction 


In this section, we further investigate the geometric structure of the feasible region. 


Definition 2.3.1 Let P’ be a nonempty convex subset of P. It is a face of P if for 
any x € P’, satisfying x € (y, z) C P, it holds that y, z C P’, where (y, z) denotes 
the open segment with endpoints y and z. 


The preceding means that a face includes the whole segment if it includes an interior 
point of any segment of P. 
The following renders a general expression of P. 


Theorem 2.3.1 The nonempty convex subset P' is a face of P if and only if there 
exists index set J’ such that 


Pl ={xeP|xj=0, jeJ'}. (2.11) 


Proof Sufficiency. Let P’ Cc P be defined by (2.11). If v € P’ is an interior point 
of segment (y, z) and y, z € P, then there exists 0 < a < | such that 


v=ay+(1—a)z. 
Hence froma > 0, 1—a>0, y,z > 0, and 
vj =ayj; + (1—a@)z; =0, jeJ, 
it follows that 


yj, zj =9, J€ J’. 
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Therefore, y, z € P’, and hence, P’ is a face of P. 
Necessity. Let P’ 4 @ be a face of P. Introduce 


J={jeAlxj=0, xe P’}. (2.12) 
Now we show that P’ is equivalent to 

P" ={xeP|xj)=0, je J}. (2.13) 
It is clear that P’ C P” C P. 

If P’ includes the origin O only, it follows that b = Oand J’ = A. And P” clearly 
includes 0, hence P’ = P”. Now assuming that P’ does not include 0 uniquely, we 
show P” c P’. 

Assume x € P”. If x = 0 (b = 0), then for any 0 4 v € P’ and 

y=20, 


it holds that 


which means that v € (y, 0). Since P’ is a face and y,0 € P, wehavex =O e€ P’. 
On the other hand, if x 4 0, ie., 


S={jeA|x;>0} 
is nonempty, then from x € P” and (2.13), it is follows that j ¢ J’ for any j € S. 
Therefore, there exists u € P’ such that uj > 0. If S includes two or more indices, 
then u, w € P’ satisfy uj, w; > 0 for any i, j € S; hence, z = u/2+w/2 € P’ 


satisfies z;,z; > 0. This means that there exists v € P’ such that vj; > 0 for all 
j € S. As for the relation between x and v, there are the following two cases only: 


(i) {7 € S| x; > vj} =. It is clear that 
z=2vu-x=vt+(u-x)eEP. 


Since P’ is a face of P, and v = z/2 + x/2 € (x, z), it holds that x € P’. 
(ii) {7 € S| x; > vj} AY. Define 


Z=x+B(v—-x), B = min{x;/(xj — vj) |xj —vj > 0,7 € S}> 1. 
It is easy to verify that z € P, and 


v=az+(1—a)x € (x, Z), 0O<a=1/8 <1. 
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In addition, P’ is a face of P, hence x € P’. Therefore, P” C P’. 
We state the following result without proof. 


Proposition 2.3.1 A subset of feasible region P is a face if and only if it is in the 
relative interior of P. 


Definition 2.3.2 A point x is a vertex of P if x € [y, z] leads tox = y or x = z for 
any y,ze€ P. 


Vertex is also termed extreme point, as it is not an interior point of any segment 
of P. It is clear that the origin is a vertex if it belongs to P, and a vertex of face is a 
vertex of P. 


Lemma 2.3.2 A point x € P is a vertex if and only if columns of A corresponding 
to its positive components are linearly independent. 


Proof It might be well to let the first s components of x be great than 0, and let the 
rest be 0. Assume that x is the subvector consisting of the first s components of x 
and A is the submatrix consisting of the first s columns of A. Then Ax = b. 

Necessity. Let x be a vertex of P. If columns of A are linearly dependent, there 
is a nonzero vector 0 such that Av = 0. Introduce 


y=kt+av, Z=*—ad. 


It is clear that for any real @ it holds that 


> 


Ay = AZ = 


Take sufficiently small a > 0, such that y,z > 0. Construct vectors y and z, so 
that the first s components of them, respectively, constitute y and z, and the others 
are 0. Then it is clear that y,z € P andx = y/2+ z/2. Thus x is not a vertex 
of P, as leads to a contradiction. Therefore, columns corresponding to all positive 
components of x are linearly independent if x is a vertex of P. 

Sufficiency. Assume that columns of A are linearly independent. If x € P is not 
a vertex, there are two points y, z € P andareala € (0, 1) such that 


x=ay+(1-a)z, 


from which it is known that the last n — s components of y and z are both 0. 
Therefore, v = x — y £ O and 


which means that columns of A are linearly dependent, as a contradiction. Therefore 
x is a vertex if columns of A are linearly independent. oO 
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In view of rank(A) = m, the preceding theorem implies that the number of 
positive components of a vertex of P is no more than m. 


Lemma 2.3.3 A point x is a vertex of feasible region if and only if it is the feasible 
basic solution. 


Proof It is from Theorem 1.5.2 and Lemma 2.3.2. oO 


Therefore, the vertex and feasible basic solution are, respectively, geometrical 
and algebraical names of the same thing. It is usually difficult to determine whether 
a point is a vertex but can determine a feasible basic solution conveniently. Recall 
that every canonical form of Ax = b corresponds to a basic solution, which is a 
feasible basic solution if it is nonnegative (Sect. 1.5). 


Lemma 2.3.4 A nonempty feasible region has a vertex. 


Proof It is clearly the case when P is a single point set. Let P be an infinite set, 
and x € P. If x is not a vertex, there are two distinct points y,z € P and a real 
a € (0, 1) such that 


x=ay+(l-—a)z=z+a(y —2Z). 


Thus, a component of x is 0 if and only if the corresponding components of both 
y, z are 0. Introduce 


T ={j€A| x; > 0}. 


Then T 4 Y, since T = J implies that x = 0 is a vertex. It might be well to assume 
that for some i € {1,...,} it holds that z; > y;, and hencei € T. This means that 


{7ET | zj-yji > OH FY. 


Assuming 
X = ay+(1—a))z, a1 = 2q/(Zqg—Yq) = min {z;/(zj—yj) | zj-Yj > 9, F © Th, 


it is easy to show that x € P and x, = 0. Thus |T| for the new x is less than that 
for the old by 1, at least. Repeating no more than n times, therefore, the preceding 
process terminates at a vertex. oO 


It is clear that feasible region P is the maximum face. The above proof produces 
a series of feasible points, corresponding to faces, each of which is a proper face 
of its predecessor until reaching a vertex, viewed as 0-dimensional face. Such a 
technique for shifting to faces of lower dimensions is useful. 

The feasible region P has a finite number of faces. If dim P’ = (dim P) — 1, 
the P’ is called facet. An 1-dimensional face is also called edge. There exist at most 


C+! edges and C?-™ = C™ vertices. 
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For any given point x and vector d ¥ 0, the set {x + ad | a > 0} is said to bea 
ray (or half-line), emanating from x along the direction of d. It is clear that a ray is 
an infinite set. 


Definition 2.3.3 A nonzero vector d is an unbounded direction if P includes rays, 
emanating from all x € P along d. 


Two unbounded directions having the same direction are regarded as the same. 


Theorem 2.3.5 A vector d is an unbounded direction of the nonempty feasible 
region if and only if 


Ad = 0, d #0, d>0. (2.14) 


Proof Sufficiency. With the assumptions, it is easy to verify that for any given x € 
P anda > 0 it holds that x + ad € P; therefore d ¥ 0 is an unbounded direction. 
Necessity. Let d 4 0 be an unbounded direction. Thus, there is x € P, satisfying 
x-+ad € P forany a > 0. Hence, from Ax = b and A(x + ad) = b, it follows that 
Ad = 0. In addition, d > 0 holds, because, otherwise, d has a negative component, 
and hence, the corresponding component of x + ad is negative whenever a > 0 
becomes sufficiently large, which contradicts x + ad € P. Oo 


Corollary 2.3.6 A nonzero vector d is an unbounded direction if P includes the 
ray, emanating from some x € P along d. 


It is clear that any nonnegative linear combination of finitely many unbounded 
directions is an unbounded direction if the combination coefficients are not all zero. 
Note that the “unbounded direction” is meaningless to an empty P. 


Theorem 2.3.7 The feasible region is unbounded if and only if it has an unbounded 
direction. 


Proof Sufficiency is clear, and it is only needed to show necessity. 
If v € P, then the translation of P, 1.e., 


C={x-v|xe P} 
clearly includes the origin, and P is unbounded if and only if C is unbounded. It 
might be well to assume that 0 € P. 
Let S’ = {x*} © P be an unbounded sequence of points. Without loss of 
generality, assume that 
I|x* || > ow as k> ow. 


Then the sequence 


Se at 
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on the unit sphere is bounded and hence has a cluster point. Letting x be its cluster 
point, then S” includes a subsequence converging to x. We might as well assume 


k k 
x” [|x|] > x. 


Now it should be shown that x is an unbounded direction of P. Let M be a given 
positive number. Since I|x* || — oo, there is a positive integer K such that IIx || > 
M or M/||x*|| < 1 whenk > K. Introduce 


yk = (M/|x* |) x*, k=K,K+1,.... 
Since P is convex, we have 0, x* € P and yk Ee (0, ay, and hence, i e€ P and 
yk — Mx whenk > K. As P is closed, it can be asserted that Mx € P. oO 


Definition 2.3.4 An extreme (unbounded) direction is not a positive linear combi- 
nation of two distinct unbounded directions. 


According to the preceding, that d is an extreme direction means that if there 
are unbounded directions d’, d” and positive numbers 01,02 > 0, satisfying d = 
o\d' + ood", then there must be d’ = od’, where o > 0. Two extreme directions 
having the same direction are regarded as the same. 

In Fig. 2.2, d', d*,d? are unbounded directions of P. d! and d? are extreme 
directions, but d? is not. 


Theorem 2.3.8 An unbounded direction is an extreme direction if and only if the 
rank of columns, corresponding to its positive components, is less than the number 
of columns by 1. 


Fig. 2.2 d!,d*,d? are 
unbounded directions. d!, d? 
are extreme directions, but d? 
not 
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Proof We might as well assume that k positive components of unbounded direction 
d correspond to the set of columns aj, ..., ax. Satisfying Ad = 0 implies that the 
set is linearly dependent. Denote the rank of the set by r; then it is clear that r < k. 
Without loss of generality, assume that the first r columns are linear independent. 
Introduce B = (aj,..., a,). It is clear that 


rank B=r<rankA=m. 


Note that k > 2 because if k = 1, it follows from Ad = O that aj = 0, a 


contradiction. 
Necessity. Assume that d is an extreme direction, but r # k — 1, that is, 
a\,...,Ax—1 are linearly dependent. Thus, there exists 


y= (y1, +++, Ye-1,0,...,0)7 #0 


such that 
k-1 
Ay = x yjaj =0. 
j=l 


Clearly, for sufficiently small 5 > 0, it holds that 
0OAd' =d+5y>=0, 04d" =d-—5Sy>=0, 


and Ad’ = Ad” = 0. Therefore, d’, d” are unbounded directions of P and hence 
not of the same direction. But d = (d’ + d”)/2, contradicting that d is an extreme 
direction. It therefore holds that r = k — 1. 

Sufficiency. Assume r = k — 1. If there exist unbounded directions d’, d’ and 
01,02 > O such that 


d= oid’ + ond", 
then zero components of d clearly correspond to zero components of d’ and d”. So, 
the last n — k components of d’ and d” are all zero. In addition, since d’, d" are 
unbounded directions, it holds that Ad’ = Ad” = 0 (Theorem 2.3.5), and hence 
that 
Bd, + dak = 0, Bdp + dj.ak = 0. 


It is clear that d,, d/ > 0. Premultiplying the two sides of the preceding two 
equalities by B™ gives 


B'Bd,+4,B'a,=0,  B'Bd, +d/B'a, =0, 
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from which it follows that 
dy = —di(B'B)'B'a,, dy, = —di'(B'B)~' Bax. 
Therefore, it holds that 
d” = (dj /d,)d', 


which means that unbounded directions d’ and d” have the same direction, and 
hence d is an extreme direction. oO 


The following reveals the relationship between the extreme direction and 1- 
dimensional face (edge). 


Theorem 2.3.9 A vector is an extreme direction if and only if it is the unbounded 
direction of an edge. 


Proof Necessity. Assume that d is an extreme direction. Based on Theorem 2.3.8, 
assume that columns corresponding to its positive components are 


a\,.--,4r,4m+1, 
where the first r < m columns are linearly independent. Thus 
d,...,dy,dm41 > 9, dpi, +++sAm, dm42,...,dy = 0. (2.15) 
Since the rank of A is m, there are m —r columns that together with a), ..., a, form 
a basis when r < m. Without loss of generality, assume that the first m columns of 
A constitute a basis, 1.e., 
B= (q,...,4r,Gp41, +++, 4m), N = {dm4i, +++, 4n}. (2.16) 
From (2.15) and Ad = 0, it follows that 
r 
Se diaj =P dmn41Am+1 = 0, 
i=1 


hence 


: 
am41 = —())Gi/dm4i)ai. 
i=1 


Assume that the basis B corresponds to the canonical form below: 


xp =b—Nxy. (2.17) 
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Since its augmented matrix (J N | b) comes from (A | b) = (B N | b) by elementary 
transformations, and a,,4, is a linear combination of aj,...,a,, the last m —r 
components of column 4,41 in the canonical form are all zero. 

Let x belong to P. If x, = 0, then x is the feasible basic solution corresponding 
to basis B. Now assume, otherwise, that x, + 0. We will show that the resulting 
B corresponds to a feasible basic solution if properly exchanging a,,,; and one of 
a\,..., yr, as Well as d-41,..., Am and dm42,..., An. 

Assume that x; > O holds forsome j € N, j 4 m+1. Reducing x; and keeping 
the other nonbasic components unchanged, we determine the corresponding value 
of xg such that x satisfies (2.17), resulting in a new solution x. For the column 
corresponding to x ;, of the canonical form (2.17), there are only two cases arising: 


(i) aj = 0. 
It is clear that x4) may decrease to 0 and associated xg remains nonnegative, 
and thus we set x; = 0. 

(ii) a; 7 0. 

If the first r components of a; are nonnegative, one of the last m — r components 
of xg decreases to 0 the earliest (the so-called blocking) as x; decreases. Assume 
that itis componenti (r+1 <i < m). Set x; to the associated value, and interchange 
aj; and a; to update B and N. By exchanging their indices at the same time, the 
nonbasic component x; of the new solution x becomes 0. 

If some of the first r components of a; are negative, then we can determine a 
o > Osuch that the first r basic components of 


X:=x+od 


are sufficiently large and all the nonbasic components remain unchanged, except for 
Xm+1, such that no broking happens to the first r components of xg as x; decreases. 
If no broking happens to the last m — r components either, we set x; = 0. If 
broking happens to component r + 1 <i < m, we set x; to the according value 
and interchange a; and a; to updated B and N. Consequently, by exchanging their 
indices, the nonbasic component x; of the new solution x is now equal to 0. 

As such, we can transform all x;, j ¢ N, j #m-+ | to 0, without affecting the 
first r indices of B. Then, if X41 = 0, we are done. 

If Xm+1 > 0, we deduce it and keep the other nonbasic components unchanged. 
Since the last m — r components of a+; are zero, the corresponding components 
of xg remain unchanged, and there will be only two cases arising: 


(i) The first r components of d+; are all nonnegative. Then it is clear that X.+1 
can decrease to 0 and according xg remain nonnegative; thus we set X41 = 0. 

(ii) Some of the first r components of G41 are negative. If the according xg 
remains nonnegative as Xm+1 decreases to 0, we set Xm+1 = 0; otherwise, if 
broking happens for component | <i <r of Xg, we set Xm+1 to the associated 
value and interchange a4 1 and a; to update B and N. Then, by exchanging 
their indices, the nonbasic component x,,41 of the new x is equal to zero. 
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Therefore, it can be asserted that the basis B, defined by (2.16), corresponds to 
the feasible basic solution x. 
Consider the following face 


P’={xeR" | Ax=b, x >0, xj =0, j=m+2,...,n} 


={x ER" | BxgtxXmpidmoi = b, XB, Xm41 = 0; x7 = 0, f=mt+2,...,n} 
={x ER" | xp t+Xm414m41 = b, XB, Xm41 20; xj =0, f=m+2,...,n}. 
(2.18) 


It is clear that x ¢ P’. Hence, from (2.15) and 
Bdg + dmn+idm+i = 0 (2.19) 
it is known that 
X+ade P’, Va>0. 


Therefore, d is an unbounded direction of P’. It is now only needed to show 
dim P’ = 1. 
For any x’ € P’ and x’ # x, introduce d’ = x’ — x. It is clear that 


eee, a aan > 0, Ce rere ae rer =0 (2.20) 
and 
Bd, + Dn dm+1 = 0. (2.21) 
From (2.19) and (2.21) it follows, respectively, that 
dz= tn B nei; d, _ —d',4 Bo am41- 


Therefore, d’ = (di41/dm+i)d, where G44 /dm+1 > 0. This implies that dim P’ = 
1. Therefore, P’ is a 1-dimensional face or edge, and d is an unbounded direction 
of it. 

Sufficiency. Assume that d is an unbounded direction of edge P’ (2.18) and hence 
satisfies (2.19). If d is a positive linear combination of unbounded directions a’, d” 
of P, then there exists a correspondence between zero components of d and of 
d', da”, and hence d’, d"” are also unbounded directions of P’. Since dim P’ = 1, in 
addition, d’ and d” have the same direction. Therefore, d is an extreme direction. 

Oo 


Lemma 2.3.10 An unbounded direction that is not an extreme one is a positive 
linear combination of two unparallel unbounded directions. 
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Proof Without loss of generality, assume that positive columns a), ..., ag corre- 
spond to positive components of an unbounded direction d, and the first r columns 
are linearly independent. As d is not an extreme direction, it holds by Theorem 2.3.8 
thatr < k —1,or 


k-r>2. (2.22) 
Introduce By = (aj,...,a,). Within the set of ag41,...,a,, determine m — r 
columns, corresponding to B} = (ax41,..-,4k+m—r), to construct basis B = 
(B,, By) (it is clear that m — r < n — k). Then the nonbasic matrix is N = 
(Ni, N2), where Nj = (ar41,..., 4k), N2 = (dk+m—r+1,-++,4n). Assume that 
B corresponds to the canonical form xg = b — Nxy. Then d satisfies 

Bidp, = —Nidy,; dy, = 0, 
or equivalently, 
dz, = —(BiB,) | Bi Nidn,, dy, = 0. 


Introduce e € R*~" and 


dp, = —(B} Bi) 'Bi Mi (dy, +ee), dg, = —(B] Bi) |B} Ni (dn, — €e). 


Letting € = (€,..., eer)! > 0, then 
dz, dy, 
d= 0 and d’= ” 
dn, +e dn, —e€E 
0 0 


satisfy d = d’/2+d" /2. Itis clear that d’, d” #4 O and Ad’ = 0, Ad” = 0. As (2.22) 
holds, it is known that there exists a sufficiently small € such that d’,d” > 0 are 
unparallel unbounded directions. oO 


Theorem 2.3.11 Jf P has an unbounded direction, it has an extreme direction. 


Proof Let d be an unbounded direction. Assume that it has k positive components 
and the rank of the corresponding columns is r < k—1.Ifr = k — 1, thend 
is an extreme direction (Theorem 2.3.8). Otherwise, by Lemma 2.3.10, d can be 
expressed as 


d=o\d' + ood", 


where d’, d" are unparallel unbounded directions, and 01, 02 > 0. Without loss of 
generality, assume that d” has at least one component greater than the corresponding 
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component of d’. Thus, the following vector 
d=d"—-a(d"-d'), a= min{d7'/(dj —d’) | dj —d; >0, j=Hl,...,n}>0 
is well-defined. Consider 

Ad =0, d>0, d#0, 


where the first two equalities clearly hold. If the third does not hold, then d” — 
a(d” — d') = 0, as leads to 


implies that d’, d” are parallel, as a contradiction. Therefore, the third equality also 
holds, and hence, d is an unbounded direction. In addition, it is known that the 
number of zero components of d is less than that of d by 1, at least. Then set d = d 
and repeat the preceding steps. It is clear that such a process can only be repeated 
finitely many times, and it terminates at some extreme direction. oO 


According to Theorem 2.3.9, extreme directions and unbounded edges of P are 
1-to-1 correspondent (extreme directions having the same direction are viewed as 
the same). As there are finitely many edges, the number of extreme directions is 
finite. 


2.4 Representation of Feasible Region 


It is now time to lay a theoretical basis for the Dantzig—Wolfe decomposition method 
for solving large-scale LP problems (Chap. 9). 


Theorem 2.4.1 (Representation Theorem of Feasible Region) Let P be 
nonempty. Assume that {u!,..., u°} is the vertex set and {v',..., v'} is the extreme 
direction set. Point x belongs to feasible region if and only if 


s t 
x= So au + > Bu 
i=1 j=l 


See, a >0, i=1,...,5, (2.23) 


Bj = 9, P= lease pe 


Proof When (2.23) holds, it is clear that x € P. So it is only needed to show 
necessity. We use an inductive method to dimensions. 
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If dim P = 0, P is a single point set, including a vertex. The conclusion holds 
clearly. Assume that it holds for dim P < k. We will show that it holds for dim P = 
k>1. 

From Proposition 2.2.2, it follows that int P #4 @. Assume x € int P, and 
consider 


x’ =x—dAtu! —x). (2.24) 


Note that u! 4 x. There are the following two cases arising: 


(i) ul—x £0. 
Determine 2 such that 


A = Xq/(ujy — Xq) = min{xj/(u; — xj) | uj) — xj > 0, f=1,...,n} > 0. 


Then, it is easy to verify that x’, defined by (2.24), satisfies x’ € P and 
x4 = 0. Therefore, x’ belongs to some proper face with its dimension less 
than k. According to the assumption of induction, x’ can be expressed as the 
sum of a convex combination of vertices and a nonnegative combination of 
extreme directions of the proper face. Since vertices and extreme directions of 
a face are also that of P, therefore, x’ can be expressed as the sum of a convex 
combination of vertices and a nonnegative combination of extreme directions 
of P, i.e., 


S| ty 
a Po hd 
x= Dou! + DB, 
i=l jel 


Bb’, = 0, (i Pere ce 


where u' and v/ are vertices and extreme directions of P, respectively. 
Substituting the preceding into 


_ 1 a 1 1 | 
nes hs [pay °° 


which is equivalent to (2.24), leads to the expression of the form of (2.23), Le., 
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(ii) u! —x <0. 
Then the point x’, defined by (2.24), is the feasible point for any A > 0; hence, 
—(u! — x) is an unbounded direction. According to Lemma 2.3.11, there is an 
extreme direction, say v!. Now take a sufficiently large jx such that 


K=ul +p! 


has at least one component greater than the corresponding component of x. 
Consequently, the point defined by 


x’ =x-—A(¥—x), A= Xq/(X%q — Xq) = min{x;/(xj; — x;) | 


xj —xj >0, j=l,...,n}>0 


is a feasible point, satisfying xq = 0. Therefore, this point belongs to some 
proper face with dimension less than k and hence can be expressed as the sum 
of a convex combination of vertices and a nonnegative combination of extreme 
directions of P. As a result, an expression of the form (2.23) can be obtained in 
an analogous manner as case (i). oO 


The preceding theorem indicates that a feasible point is the sum of a convex 
combination of vertices and a nonnegative combination of extreme directions and 
vice versa. In particular, the following is a direct corollary. 


Corollary 2.4.2 Let the feasible region be bounded. A point is feasible if and only 
if it is a convex combination of vertices. 


2.5 Optimal Face and Optimal Vertex 


To begin with, we present a basic result without the proof (see, e.g., Rockafellar 
1997). 


Theorem 2.5.1 Let x be a boundary point of convex set S. Then there exists a 
hyperplane including x and partitioning the total space into two half-spaces, one 
of which includes S. 


The hyperplane involved in the preceding theorem is said to be supporting 
hyperplane of S. That is to say, there is a supporting hyperplane through every 
boundary point of a convex set. Although the result applies to any convex set, we 
are only concerned with the feasible region P, in particular. 

Supporting hyperplane of P is closely related to its face, as the following reveals. 


Lemma 2.5.2 The intersection of P and a supporting hyperplane is a face. 


Proof Assume that H = {x € R” | a'x = n} is a supporting hyperplane of P, and 
P' = PH. Without loss of generality, let a'x < n hold for all x € P. 
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Assume that v € P’ is an interior point of segment (y, z), and y, z € P, ie., 
v=ay+(1-a)z, (2.25) 


where 0 < a < 1. Note that P’ is a nonempty convex set. 

It is only needed to show y, z € H, as leads to y, z € P’; hence, P’ is a face of 
P. 

Assume y, z ¢ H. Then it holds that 


a'y <7, a'z <7. 


Multiplying the preceding two formulas, respectively, by a > 0 and 1 — a > 0 and 
then adding the results give 


a (ay +(1—a@)z) < natn —a@) =n, 


combining which and (2.25) leads to v ¢ H, and hence v ¢ P’. This contradicts the 
assumption v € P’. Therefore, at least one of y and z belongs to H. 
Without loss of generality, assume z € H. Then, it follows from (2.25) that 


y= (l/a)u+ (l — 1/a)z, 


implying that y is in the straight line through v and z. Moreover, z, v € H and H is 
a supporting hyperplane, therefore y € H. oO 


Definition 2.5.1 The optimal (solution) set is the set of all optimal solutions. 


Lemma 2.5.3 Let f* be the optimal value of the standard LP problem. The set F 
is the optimal set if and only if it is the intersection of P and the optimal objective 
isosurface, i.é., 


H={x eR" | c'x =f". (2.26) 


Proof Assume F = PH. 

It is clear that any optimal solution x € P satisfies c'x = f*, implying x € H, 
hence x € F.. Therefore, F is the set of optimal solutions. If F is the set of optimal 
solutions, conversely, then c'x = f* holds for any x € F C P. Therefore, x € H, 
and hence, x C P/N H. oO 


The # is clearly a supporting hyperplane of P, referred to as the objective 
supporting hyperplane. 

A face is optimal if its elements are all optimal solutions. A vertex is optimal if 
it is an optimal solution. 


Lemma 2.5.4 [f there exists an optimal solution, there exists an optimal face. 
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Proof According to Lemma 2.5.3, a nonempty set of optimal solutions is the 
intersection of feasible region P and the objective isosurface H. Therefore, it is 
an optimal face, associated to Lemma 2.5.2 and the definition of an optimal face. 

oO 


Theorem 2.5.5 If there is a feasible solution, there is a feasible basic solution. If 
there is an optimal solution, there is an optimal basic solution (vertex). 


Proof By Lemmas 2.3.3 and 2.3.4, it is known that nonempty feasible region has 
a feasible basic solution. By Lemma 2.5.4, if there is an optimal solution, there is 
an optimal face, which is a nonempty polyhedral convex set and hence having an 
optimal convex or optimal basic solution. Oo 


A vertex is a 0-dimensional optimal face. In general, there exist optimal faces of 
higher dimensions. The optimal set is just the largest optimal face, i.e., the optimal 
face of the highest dimension. The following is obvious. 


Proposition 2.5.1 /f there exists an optimal solution, there exists the largest optimal 
face. 


2.6 Graphic Approach 


An LP problem of 2-dimension can be solved via a graphic approach. To do so, let 
us return to Example 1.2.2. The area of polygon OABCD in Fig. 2.3 is the feasible 
region (ignore the straight line x + 2y = 10, at the moment). It is required to 


2x+5y=0 E 


Fig. 2.3. Graphic solution to Example 1.2.2 
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determine a point over the area such that the objective function reaches the highest 
value at the point. 

In the figure, equation 2x + 5y = 0 of the isoline of the objection function 
corresponds to the straight line OF, going through the origin, all points on which 
correspond to the same objective value 0. The line’s slope, the tangent of the angle 
between it and the x axis, is —2/5 = —0.4. Therefore, the corresponding isoline 
parallel shifts up to the upper-right side as the objective value increases from 0. 
The points on the intersection of the line and the area OABCD are all feasible 
ones. The parallel shifting should be carried out as far as the intersection remains 
nonempty while achieving the largest possible objective value. It is seen that the 
farthest shifting line is BF though vertex B. The optimal set, i.e., the intersection 
of the line and the feasible region, includes a single vertex B, corresponding to 
the optimal basic solution. Consequently, problem (1.6) is solved after measuring 
coordinates of point B and calculating the associated objective value. 

If the figure or measurement is not accurate, however, the outcome may involve 
unacceptable errors. For a graphic approach, therefore, it might be desirable to 
use coordinate paper together with simple calculating. Once B is known to be the 
optimal vertex, its coordinates can be obtained by solving the following system of 
equations: 

lila = 12 (2.27) 
y= 3 


from which the optimal basic solution x = 1.5, y = 3 follows with optimal value 
f = 18. That is to say, the manufacturer should arrange production of 1.5 thousand 
laths and 3 thousand sheep piles daily, gaining 18 thousand dollars profit. 

Although the graphic approach does not apply to the case of n > 3, it may give 
us some useful enlightenment that is the theme of the next section. 


2.7 Heuristic Characteristic of Optimal Solution 


From the graphic approach, it is seen that the optimal solution to the LP problem 
is attained at a vertex. It is imaginable that if the feasible region has an edge 
passing through a vertex, which is parallel to objective isolines, then the whole edge 
corresponds to the optimal set with the same optimal value. 

Therefore, the key is how to determine lines intersecting at the optimal vertex. In 
other words, it is only needed to know which inequalities are binding at the optimal 
solution. Once these inequalities are known, what left to do is just to solve a linear 
system. 

It is observed that normal directions (pointing to the interior of the feasible 
region) of lines AB and BC, intersecting at the optimal vertex B, form the maximum 
angles with the parallel shifting direction of the isoline B F among all lines. 
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Now turn to the more general minimization problem 


min f =c'x 
s.t. Ax > b, ee) 
where A € R”*", CER", DER”, m> 1. 

We can make a similar image in R” space. Now examine angles between normal 
directions and parallel shifting direction of the isosurface. For minimization, the 
shifting direction is the negative gradient of the objective function. This leads to the 
following plausible statement (Pan 1990): 


Proposition 2.7.1 (Heuristic Characteristic of Optimal Solution 1) Gradients of 
constraints binding at optimal solution tend to form the maximum angles with the 
negative objective gradient. 


Thus, we need to quantify sizes of angles. Denote the ith row vector of A by 
a}. The cosine of the angle between the ith constraint gradient and the negative 
objective gradient is then 


cos(a;,c) = —@) c/(||a;I|Icll). 


For simplicity, we ignore factor 1 /||c|| and introduce the following: 


Definition 2.7.1 The pivoting-index of the ith constraint is defined as 
_ _-T Ass 
a; = —a}c/|lai I). (2.29) 
Then, we are able to compare angles with pivoting-indices, and Proposi- 


tion 11.1.2 can be transformed into: 


Proposition 2.7.2 (Heuristic Characteristic of Optimal Solution 2) The gra- 
dients of constraints binding at an optimal solution tend to have the minimum 
pivoting-indices. 


Example 2.7.1 Investigate problem (1.6) via pivoting-indices: 


min f = —2x —5y 


s.t. —2x —3y > —-12 
—-x-y> -5 (2.30) 
= 


x,y> 0. 
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Answer Calculate pivoting-indices of the constraints, and put them in the 
following table in the order of increasing pivoting-indices: 


Constraints Qj 
2x —3y > —-12 5.26 
-y>-3 —5.00 
x-y>-5 4.95 
x>0 2.00 
y>=0 5.00 


From the preceding table, it is seen that —2x — 3y > —12 and —y > —3 are 
the two smallest pivoting-indexed constraints. Thus, the two are binding constraints 
at the optimal solution. This leads to the system (2.27) again, coinciding with the 
outcome of the graphic approach. 


If Proposition 11.1.2 were true in general, solving an LP problem amounts 
to solving a system of linear equations by O(m*) basic arithmetics (!) But the 
characteristic of the optimal solution is heuristic only. Counterexamples can be 
easily constructed. If adding a constraint x +2y < 10 to (1.6) (see Fig. 2.3), e.g., the 
added constraint is not binding at the optimal solution, although its gradient forms 
the maximum angle with the objective gradient (with pivoting-index —5.37), since 
adding is a redundant constraint. 

Someday in the Fall of 1986 when the author was visiting the Mathematics 
Department of University of Washington, after he talked with Professor Rockafellar 
about his idea on the characteristic of an optimal solution, the latter quickly showed 
a counterexample by sketching on a piece of paper. 

For all that, Proposition 11.1.2 might still serve as a clue to the optimal solution. 
Such a trick will be referred to as the most-obtuse-angle heuristic in this book. 

In fact, in some cases, it is possible to detect the unboundedness of a problem 
simply by signs of pivoting-indices. 


Theorem 2.7.1 Assume that the feasible region is nonempty. If the pivoting-indices 
of constraints are all nonnegative, then the problem (2.28) is unbounded. 


Proof By contradiction. Assume that the problem is bounded under the assump- 
tions. It follows that 


clv <0, Vue{verR” | Av>O}. (2.31) 
In fact, there is v such that 


Av>0, clu>0. (2.32) 
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Thus, for any a > 0 and feasible solution x, vector x = x + av satisfies Ax > b, 
that is, v is an unbounded direction. Further, it is known from (2.32) that 


clx =clx+aclv > oo, (asa — oo), 
which contradicts that (2.28) is upper bounded. Therefore, (2.31) holds. 
Thus, according to Farkas’ Lemma 2.0.3, there is y > O such that 


-—c= A'y, 


premultiplying which by —c" gives 


Tr 


0<cle=—y'Ac. 


Thus Ac > 0 follows from the non-negativeness of pivoting-indices. This implies 
that the right-hand side of the preceding is less than or equal to 0, as is a 
contradiction. Therefore, the problem is unbounded. oO 


An alternative proof of the preceding theorem is via showing c to be an 
unbounded direction. The following corollary gives a necessary condition for the 
existence of an optimal solution. 


Corollary 2.7.2 If there is an optimal solution, then there is at least one constraint 
bearing a negative pivoting-index. 


Example 2.7.2 Investigate the following LP problem by pivoting-indices: 


min f=-x-y | qj 
st. 2x—y>-3 |0.71 
—3x+4y > 4 |0.20 
2x +2.5y>5 \1.41 
—2x + 4y > —8)0.45 
y>1.2 1.00 
x>0 1.00 
y>0 1.00 


(2.33) 


Answer Calculate indices of the constraints, and fill in the right-hand side of the 
preceding table. According to Theorem 2.7.1, the problem is unbounded since all 
indices are nonnegative (see Fig. 2.4). 


2.8 Feasible Direction and Active Constraint 


Methods for solving mathematical problems fall into two categories: direct and 
iterative methods. The latter produces a sequence of points by iterations, offering 
an exact or approximate solution. Methods presented in this book belong to the 
iterative category. 
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Sy 


—x-—y=0 


Fig. 2.4 Unbounded problem 


Line search is the mostly used iterative approach in optimization. At a current 
point x, in each iteration, a new point Xx is determined along a ray starting from x 
along a nonzero vector d, i.e., 


R=E+ad, (2.34) 


where d is referred to as search direction, a > 0 as stepsize. Once the two are 
available, x can be calculated and then one iteration is complete. Repeating this 
process yields a sequence of points, until a solution is reached. Formula (2.34) is 
referred to as line search or iterative scheme. 

The determination of a search direction is crucial. In the presence of constraints, 
d should be such that the intersection of the ray (2.34) and the feasible region is 
nonempty. To be precise, we introduce the following concept: 


Definition 2.8.1 Let P be the feasible region. Assume that x €¢ P, d # 0. If there 
exists @ > O such that 


X+ade P, Va é[0, a], 


d is a feasible direction at x. 


The preceding is relevant to general constrained optimization problems, includ- 
ing LP problems. The following are some instances, in conjunction with feasible 
region P. 


Example 2.8.1 In Fig.2.5, x is an interior point of P, and hence any direction is 
feasible at it. 
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Fig. 2.5 Any direction is 
feasible at x 


Fig. 2.6 d>, d* are feasible 
at x, but d', d? not 


Fig. 2.7 d*,d>, d°, d’ are 
feasible at x, but d!, d*, d? 
not 


Example 2.8.2 In Fig.2.6, x is a boundary point of P. It is seen that d?, d+ are 
feasible directions at x, but d!, d? are not. 


Example 2.8.3 In Fig. 2.7, x is a vertex of P. Any direction, e.g., d* or d>, within 
the angle area between d® and d’ (which are along the two edges of P, respectively) 
is feasible at x. The vectors d!, d?, d? are not feasible (Fig. 2.4). 
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Let d be a feasible search direction. It is possible to maintain feasibility of new 
iterate x. In order for X to be close to an optimal solution, it is needed to take into 
account the objective function. To this end, consider the following problem: 

min f =c'x 
T 


i 


(2.35) 


S.t. ax > bj, i=1,...,m, 


where m > n, and whose feasible region is 
= n T i 
P={xeR® | a,x >bj, i=1,...,m}. 


Definition 2.8.2 The vector d satisfying condition c'd < 0 is a descent direction. 
If d is also a feasible direction at x € P, it is a feasible descent direction at x. 


It is clear that d is a feasible descent direction at x if and only if it is a feasible 
direction and forms an obtuse angle with the objective gradient c. Once such a 
direction d is available, a stepsize a > 0 can be determined, and hence a new 
iterate X € P, obtained by (2.34), corresponding to a smaller objective value. Then, 
one iteration is complete. 

It is noticeable that not all constraints affect the determination of a feasible 
descent direction at a current point. 


Definition 2.8.3 A constraint that is violated, or satisfied as an equation, by the 
current point is an active constraint. 


Aimed at a feasible point, the active constraint is usually defined as one satisfied 
as equality at the point. But it seems to be useful to include an infeasible point by 
regarding a constraint violated as active. 

Let us bring up (2.35) as an example. If a} x = b, then a} x > bis active at x; 
if, otherwise, a} x > b, then a} x > b is not active at that point. So, the current 
point is on a boundary of an active constraint. From the simple instances given in 
the preceding figures, it is seen that a feasible descent direction can be determined 
by taking into account active constraints only. 

In practice, the preceding definition for an active constraint is not very satis- 
factory since a point close to some boundary would lead to a too small stepsize. 
Therefore, Powell (1989) proposes the so-called €-active constraint, where € is a 
small positive number. For example, if ae —b <e, then ais > bis an €-active 
constraint at x; whereas it is not if ae —b>e. 

The LP problem can be solved by the so-called active set method, which is often 
used for solving nonlinear programming problems though some scholars prefer it 
to be an LP problem solver (e.g., Fletcher 1981 and Hager 2002). We outline the 
method in conjunction with the problem (2.35) in the remainder of this section. 

Assume that the current vertex x is a unique solution to the linear system below: 


T. 


a; x = bi, ic A, 
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where A is called the active set, consisting of n indices of (total or part) active 
constraints, with linearly independent gradients a;. If x is judged to be optimal under 
some criterion, we are done; otherwise, some index p € A is selected such that the 
n — | equalities 


a} x = bi, i € A\{p}, (2.36) 


determine a descent edge (1-dimensional face). 
In fact, since the rank of the coefficient matrix of (2.36) is n — 1, the associated 
homogeneous system 


ajd=0, i € A\{p}, 
has a solution d such that 
d #0, cld <0. 
It can be verified that for wa > 0, all points on the ray 
X=X+ad 


satisfy (2.36). Under the condition that the objective value is lower bounded, the 
following stepsize is well-defined: 


@ = (by — aj X)/ajd = min{(bj — a) X)/a)d | ajd <0, i ¢ A}= 0. 


In fact, such an @ is the maximum stepsize possible to maintain the feasibility of x. 
Since ay x = bq, the ayx > bg is an active constraint at x. 

Consequently, setting x = X and redefining A = A\{p} U {q} complete an 
iteration of the active set method. If ~ > 0, the new vertex corresponds to a lower 
objective value. Note however that if there are multiple active constraints at the 
current vertex x, then a defined by (2.36) would vanish so that the descent edge 
degenerates to a vertex. That is to say, the “new vertex” is just the old, though set 
A changes. Such a vertex is called degenerate. In Fig. 2.7, e.g., x is a degenerate 
vertex, at which the three edges along d?, d®, d’ intersect. 

The simplex method can be viewed as a special scheme of the active set method, 
as will be discussed in Sect. 3.6. In history, the former emerged before the latter 
from another path by fully taking advantage of the linear structure. 


References 


Fletcher R (1981) Practical methods of optimization. Volume 2: constrained optimization. Wiley, 
Chichester 


References 55 


Farkas J (1902) Uber die Theorie der Einfachen Ungleichungen. Journal fiir die Reine und 
Angewandte Mathematik 124:1-27 

Hager WW (2002) The dual active set algorithm and its application to linear programming. Comput 
Optim Appl 21:263-275 

Pan P-Q (1990) Practical finite pivoting rules for the simplex method. OR Spektrum 12:219-225 

Powell MJD (1989) A tolerant algorithm for linearly constrained optimization calculations. Math 
Program 45:547-566 

Rockafellar RT (1997) Convex analysis. Princeton University Press, Princeton 


Chapter 3 ®) 
Simplex Method crests 


The simplex method is the most famous and widely used method for solving LP 
problems. Since created by George B. Dantzig in 1947, it has been dominating this 
field for more than 70 years. This chapter presents this method, including tableau as 
well as revised simplex algorithm, starting-up, degeneracy, cycling, and finite pivot 
rule. We conclude with some remarks in the final section. 

The basic idea behind the simplex method seems to be simple. It moves from 
a vertex to an adjacent vertex, while improving the objective value, until reaching 
an optimal vertex. This is based on Theorem 2.5.5, guaranteeing the existence of 
an optimal basic solution if any. Since the feasible region has a finite number of 
vertices, it seems to be natural to hunt among vertices. 

The idea may be traced back to as early as Fourier (1823). It was materialized 
algebraically by Dantzig (1951a). In this chapter, the simplex method will be 
presented in a tableau form first, and then it is revised to a more applicable version. 
Related topics are also discussed, such as how to get the method started, the 
finiteness, computational complexity, etc. The final section will make remarks on 
the simplex method. 


3.1 Simplex Algorithm: Tableau Form 


We begin by introducing the so-called simplex tableau for problem (1.10). In 
Sect. 1.5, we already obtained the canonical form 1.5.1 of its constraint system, 
without touching the objective function at all. Now we move the objective function 


xX, + 2x2 -x4+x5-— f =0, 
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where f is called objective variable, to the bottom of the constraint system, so that 
the tableau becomes 


X1 x2 X3 X4 x5 
1 2/11 —4/11 
1 15/11 —19/11 
1 1/11 9/11 
1 2, -1 1 


Then we eliminate the nonzero entries, corresponding to pivot columns, associ- 
ated with x1, x2, x3), in the bottom (objective) row. To do so, add —1 times of the 
first row to the bottom row at first: 


mal x2 X3 x4 x5 
1 2/11 —4/11 
1 15/11 —19/11 

1 ll 9/11 

2 —13/11 15/11 


Xx) x2 X3 X4 X5 
1 2/11 —4/11 
1 15/11 19/11 


1 4/ll 9/11 
—43/11 53/11 


where the northwest corner is the unit matrix, corresponding to zero entries in the 
bottom row. Thereby, the tableau offers not only a basic solution 


xp=7/ll, x. =14/1l, x3=9/11, x4 =2x5=0, (3.1) 


but also a reduced form of the objective function over the feasible region. Clearly, 
the solution is a feasible basic solution with objective value 35/11, which is equal 
to the negative southeast corner entry. 

It cannot be asserted that the feasible basic solution given by (3.1) is optimal 
because the reduced cost with x4 is negative. In fact, as the value of x4 increases 
from 0, while that of xs is fixed at 0, the objective value can reduce further. 
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The new value of x4 should be as large as possible, so that the corresponding 
objective value becomes as low as possible, subject to 


x T/1l — (2/11)x4 > 0, 
xq = 14/11 — (15/11)x4 > 0, (3.2) 
x3= 9/11—(1/11)x4 > 0, 


and keeping nonnegativity of values of x), x2, x3. The preceding inequalities are 
equivalent to 


7/2 
14/15 
9. 


tal 
- 
IA IA IA 


The preceding determines the maximum possible value taken by x4, i.e., 
x4 < min{7/2, 14/15, 9} = 14/15, 
substituting which into (3.2) gives the new feasible solution 
® = (7/15, 0, 11/15, 14/15, 0)7. (3.3) 
The associated new objective value f = —7/15 is less than 35/11. 
The related new simplex tableau can be obtained by taking entry 15/11 at row 


2 and column 4 as the pivot. First, multiply row 2 by 11/15 to turn the pivot to 1, 
leading to 


Bal x2 x3 X4 X5 f RHS 
1 : 2/11 4/11 | TAL 
11/15 1 19/15 14/15 

1 1/11 9/11 9/11 
ait. S37 [fai | 3571 


Then add —2/11, —1/11, and 43/11 times of row 2 to rows 1, 3, and 4, respectively, 
giving the new simplex tableau 


Xx] x2 X30°«OX4 x5 f | RHS 
1 -2/15 —2/15 T/15 
11/15 1 —19/15 14/15 
-1/15 1 14/15 11/15 
43/15 —2/15 —l T/15 


which clearly corresponds to the new feasible basic solution (3.3). 
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As the reduced cost with x5 in the objective line is still negative, it cannot be 
asserted that the new solution is optimal. Similarly as in the previous step, we 
consider the following set of inequalities: 

x1 7/15 + (2/15)x5 => 0 
x4 = 14/15 + (19/15)x5 > 0 
x3 = 11/15 — (14/15)xs5 = 0. 


Since coefficients of x5 in the first two inequalities are positive, the values of x1 
and x4 remain nonnegative as x5 increases from 0, while x2 is fixed on zero. It is 
therefore only needed to consider the third inequality associated with the negative 
coefficient of x5. Setting x3 = O in the third equation gives x5 = 11/14, leading to 
the feasible basic solution 


¥ = (4/7, 0, 0, 27/14, 11/14)7, (3.4) 


with objective value f = —4/7, which is lower than —7/15. 

Taking the entry 14/15 at row 3 and column 5 as the pivot, the corresponding 
simplex tableau can be obtained by entering x5 to and dropping x3 from the basis. 
To do so, multiplying row 3 by 15/14 gives: 


nal x2 X3 X4 x5 f RHS 
1 —2/15 —2/15 T/15 
11/15 1 —19/15 14/15 
-1/14 15/14 1 11/14 
43/15 —2/15 -1 T/15 


Then adding 2/15, 19/15, and 2/15 times of row 3 to rows 1, 2, and 4, 
respectively, leads to 


xX] x2 X3 X4 X5 f RHS 

1 —1/7 1/7 4/7 
9/14 19/14 1 27/14 
—-1/14 15/14 1 11/14 
20/7 1/7 -1 4/7 


with reduced costs in the bottom row all nonnegative. As will be shown a little later, 
it is now can be asserted that the final feasible basic solution is optimal, as shown 
by (3.4), and we are done. 
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Table 3.1 Simplex tableau ; , : ; i : 
able 3.1 Simp Xj jn Nim Xin X jin f RHS 
1 A ingt 77 A jn by 
1 A jngi 7" jn by 
1 Am Jm+i eet Am Jn bm 
igi 2 2 


In the preceding, we transformed the tableau iteration by iteration until reaching 
an optimal solution. Such types of tableaus are called simplex tableaus, each of 
which involves a unit matrix. Table 3.1 gives a general form of the simplex tableau:! 

Let us return to the standard LP problem (1.8) and give the formal definition of a 
basis with respect to the LP problem. 


Definition 3.1.1 Given a partition A = (B, N), where B € R™*” has full rank m. 
The submatrix B is termed basis, and N termed nonbasis. 


Without confusion, B and N also denote the corresponding index sets of columns 
of the basis and nonbasis, respectively. 
Then, the problem (1.8) can be put into the following tableau: 


xp xn  f | RHS 


B N b 
os el —1 


It is usually assumed that the basic and nonbasic index sets are as follows: 
B={ji,.--,Jm}; N=A\B, (3.5) 


where jj, i = 1,...,m is the index of the basic variable in the ith row of the 
tableau. 

Assume that through some elementary transformations, the preceding tableau 
becomes the simplex tableau 3.1, which can be concisely expressed as 


xp xn  f | RHS 
I oN b (3.6) 


zi —-1| —f 


'Tt is always possible to arrange the m x m unit matrix at the northwest corner of the tableau 
by column and row permutations. Practically, however, it is not necessary, and the submatrix 
corresponding to basic variables is usually a permutation matrix. 
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The preceding simplex tableau is frequently used thereafter, exhibiting the corre- 
sponding basic solution 


Xp =b, xn =0, 


and the negative of the end entry in the bottom line is equal to the associated 
objective value. The zy in the bottom line is termed reduced costs/gradient. 

If xg > O, the solution/tableau is said to be feasible basic. If the objective 
function attains the minimum value over the feasible region, the solution/tableau 
is said to be optimal basic. 


Lemma 3.1.1 Jf the reduced costs are all nonnegative, the feasible simplex tableau 
is optimal, giving an optimal basic solution. 


Proof The simplex tableau results from a series of elementary transformations and 
hence is equivalent to the original problem. Its bottom row represents the equation 


f= f +iyxn. (3.7) 


Assume that x is any feasible solution with objective value f . Substituting it 
into (3.7) together with zy > 0 and x > 0 leads to 

f=ftiniv=f, 
which implies that x is an optimal basic solution. oO 


The reduced costs are often called check numbers, as their sign can be used to 
judge the optimality of a feasible simplex tableau. Usually, there are negative check 
numbers in the tableau. 


Lemma 3.1.2 Assuming that Zq < 0 holds for some q € N and that 
dig < 9, i=1,...,m, (3.8) 


then the LP problem is (lower) unbounded. 


Proof The simplex tableau is associated with the constraint system 


Setting x; = 0, j ¢ N, j # q in the preceding and combining the result with the 
nonnegative constraints give 


xj, = bj — Gigxq = 0, i) ree 7m (3.9) 


Ji 
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It is known from (3.8) that the set (3.9) of inequalities hold for all x7 = a => 0, 
associated with feasible value 


f=ftaz, (3.10) 


which, since Zg < 0, can be arbitrarily low as a increases. Therefore, the problem 
is lower unbounded. oO 


If (3.8) does not hold, then the value that the nonbasic variable x, takes on will 
be restricted by set (3.9) of inequalities. It is verified that the following rule gives 
the maximum possible value a. 


Rule 3.1.1 (Row Pivot Rule) Determine a row index p and stepsize a such that 
a = bp/apg = min{b; /ajq | dig > 0, i=1,...,m} > 0, (3.11) 


which is often called the minimum ratio. 
Setting x, = a in the equation of (3.9) gives a new feasible basic solution, i.e., 
£j, = bi — adj g, i=l,....m a=, JEN, J FG; Kg = a. 
(3.12) 


Taking dpq as pivot, the associated simplex tableau is obtained by multiplying 
row p by 1/dpq to convert the pivot into 1, adding —a;, times of row p to rows 
i=l,...,m, i p, and —Z, times of it to objective row. The iteration ends with 
(B, N) updated by exchanging j, and q. 

It is seen from (3.10) that the objective value decreases strictly if a > 0, and no 
actual decrease is possible if a = 0. 


Definition 3.1.2 If some components of b are equal to zero, then the associated 
feasible basic solution/tableau is degenerate. 


In degeneracy case, the stepsize a defined by (3.11) would vanish, and hence, 
the objective function remains unchanged (see (3.10)). That is to say, the associated 
“new solution” is actually the same as the old, though the basis is changed. 

In general, there are multiple choices for g, as any column index j with negative 
z; is eligible to be chosen. Dantzig’s original rule (1947) is as follows. 


Rule 3.1.2 (Dantzig’s Column Rule) Select a column index q such that 


E in Z;. 3.13 
qd ee ( ) 
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Thus, this rule selects the column with the most negative reduced cost as the pivot 
column.” For a unit increment of the nonbasic variable Xq, this choice leads to the 
maximum possible amount of decrease in the objective value. 

The overall steps are summarized into the following algorithm (Dantzig 1948, 
1963). 


Algorithm 3.1.1 (Simplex Algorithm: Tableau Form) Initial: a feasible simplex 
tableau of form (3.6). This algorithm solves the standard LP problem (1.8). 


Determine a pivot column index q € arg MinjeNn Z;. 

Stop if Zq = 0. 

Stop ifl={i=1,...,m|aig>O}=H. _ 

Determine a pivot row index p € arg minjes bj /diq. 

Convert Gp q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

6. Go to Step 1. 


MW ARWNE 


Theorem 3.1.3. Under nondegeneracy assumption, the simplex algorithm termi- 
nates either at: 


(i) Step 2, generating an optimal basic solution; or at 
(ii) Step 3, detecting lower unboundedness of the problem 


Proof Under nondegeneracy assumption, stepsizes are all positive in the solution 
process, and hence, Algorithm 3.1.1 generates a sequence of feasible basic solutions 
with the objective value decreases strictly from iteration to iteration. This means 
that any basic solution can only appear once at most. Thus, the infiniteness of the 
process implies that there are infinitely many feasible basic solutions, which is a 
contradiction. Therefore, Algorithm 3.1.1 terminates. 
The meanings of the exits of Algorithm 3.1.1 come from Lemmas 3.1.1 and 3.1.2. 
oO 


Proposition 3.1.1 A leaving column does not enter the basis in the next iteration. 


Proof Since an entering column corresponds to a negative reduced cost and the 
pivot determined is positive (see (3.11)), a leaving column must correspond to a 
positive reduced cost after the elementary transformations performed and hence 
never enters the basis in the next iteration. Oo 


On the other hand, however, it is not difficult to construct an instance such that a 
column just entering basis leaves it immediately. Moreover, since practical problems 
are always degenerate, termination of the simplex algorithm 3.1.1 is not guaranteed, 
that is, the possibility is not ruled out that indices enter and leave basis infinitely 
many times. There are a few instances that cannot be solved by the simplex method 
(see Sects. 3.4 and 3.5) although this hardly happens in practice. 


? Any choice is eligible if there is a tie when there are multiple most negative reduced costs. 
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The simplex tableau is just a concise expression of the LP problem. Representing 
the same original problem, all resulting tableaus in the solution process can be 
viewed as equivalent. Recursive formulas between a tableau and its predecessor 
are listed below: 


1. The objective row 


B = -2y/ipg 
f= f — Bbp, 
Zj =Zj+Bapj, JEN, (3.14) 
ee bee a 
a Oi=1,...,m,iF p. 
2. The right-hand side 
a = by/apq, 
A bj —aajgi=1,...,m, iF p, (3.15) 


bi = : 
a i=p. 


3. Entries of the constraint matrix 


0, t=1,....m,tA¢p; j=q. 
a,j = I = = = ee : . 
Qj — (Gp j/Apqg)atg,t=1,...,.m,tApriIEN, j#a, 
Ap j/Apq: t=p, JEN, j#q. 
0, t=1,....m;i=l,....miAp, iFft, 
Ga 1, t=i=1,...,.m, iF p, 


—A1q/Apg, t=1,...,m,t A pr,i=p, 
Vere tie 
(3.16) 


Example 3.1.1 Solve the following problem by Algorithm 3.1.1: 


min f = —4x1 — 3x2 — 5x3 

s.t. 2x7 + x2 + 3x3 + x5 =15 
Xp +x2+%34+ x4 = 12 
—2x, + x2 — 3x3 +4x7=3 
2x1 + x2 + x6 = 


xj>0, j=l,...,7. 
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Answer Initial: The following feasible simplex tableau can be directly obtained 
from the problem: 


x1 x2 X3 X44 %X5 XG RHS 
2: 1 3* 1 15 
1 1 1 1 12 
—2 1 =3 3 
2 1 1 9 
—-4 -3 -5 


Iteration 1: 


. min{—4, —3, -5} = -—5 <0, g =3. 

.1={1, 23490. 

. min{15/3, 12/1} = 15/3, p= 1. 

. Take 3 in row | and column 3 as the pivot (marked by “x,” the same below). 


aA BWR 


Multiply row 1 by 1/3, and then add —1; 3;5 times of row 1 to rows 2, 3,5, 
respectively: 


X] x2 x3 x4 x5 X6 
2/3 1/3 1 1/3 
1/3 2/3 1 -1/3 
2* 1 
2 1 1 
207s. 473 5/3 


Iteration 2: 


. min{—2/3, —4/3, 5/3} = —4/3 < 0, q =2. 

.1={1,2,3,4454¢. 

. min{5/(1/3), 7/(2/3), 18/2,9/1} = 9/1, p =3. 

. Multiply row 3 by 1/2, and then add —1/3, —2/3, —1, 4/3 times of row 3 to 
rows 1, 2,4, 5, respectively: 


an BW ee 


Pal X2  =X3 x4 x5 X6 
2/3 1 1/6 
1/3 ii -2378 
1 1/2 
2* 12 > 44 
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Iteration 3: 


. min{—2/3, 7/3, 2/3} = —2/3 <0, gq=1. 

.1={1,2,44 49. 

. min{2/(2/3), 1/(1/3), 0/2} = 0, p = 4. 

. Multiply row 4 by 1/2, and then add —2/3; —1/3; 2/3 times of row 4 to rows 
1, 2, 5, respectively: 


On BB We 


x1 x2 X3 X4 x5 X6 x7 RHS 
1 ia =i 2 
{ =< =176 14 | 1 
1 1/2 1/2 9 
1 -1/4 1/2 -1/4 


13/6 1/3 1/2 | 37 


Now all reduced costs in the preceding tableau are nonnegative, and hence, we 
have following optimal basic solution with optimal objective value: 


x* = (0,9, 2, 1,0, 0, 0), f* = 37. 


It is seen from the preceding example that the tableau in the second iteration 
already attained the basic optimal solution, but an additional iteration was per- 
formed, due to the existence of a negative reduced cost. This occurred because of 
degeneracy, leading to a zero stepsize. Therefore, the optimality condition that the 
reduced costs are all nonnegative is only sufficient, not necessary. 

The following indicates that the condition is necessary only if the nondegeneracy 
is assumed. 


Proposition 3.1.2 [fan optimal basic solution is nondegenerate, the reduced costs 
in the associated simplex tableau are all nonnegative. 


Proof Assume that there are some negative reduced costs in tableau 3.1. Without 
loss of generality, assume Z, < 0. If (3.8) holds, then unboundedness of the problem 
follows from Theorem 3.1.2, which contradicts the presence of the optimal solution; 
if otherwise, (3.8) does not hold, it is known from the nondegeneracy assumption 
that a > O, and hence, there is a feasible value strictly less than the optimal value, 
as is a contradiction. Therefore, the claim is valid. oO 


The following result concerns the case of multiple optimal solutions. 


Proposition 3.1.3 If the reduced costs are all positive, there is a unique optimal 
solution to the LP problem. If an optimal basic solution is nondegenerate and there 
exists a zero-valued reduced cost, then there are an infinite number of optimal 
solutions; in the case when the feasible region is bounded, there exist multiple basic 
optimal solutions. 
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Proof We prove the first half first. Assume that the reduced costs in an optimal 
tableau are all positive, corresponding to optimal basic solution x. For any feasible 
solution X > 0 different from x, there is an index s € N such that x, > 0 (otherwise, 
the two are the same ). Therefore, substituting x to (3.7) leads to 


f=f+) tas, 


JEN 


which implies that x is not optimal, as is a contradiction. Therefore, there is a unique 
optimal solution. 

To prove the second half, assume that a tableau, say 3.1.1, gives a nondegenerate 
optimal basic solution and has some zero reduced costs. Without loss of generality, 
assume Z, = 0. If (3.8) holds, then inequalities of the right-hand side of (3.9) 
hold for any x7 = a > 0, that is, there are infinitely many feasible solutions, 
corresponding to the same optimal value f (see (3.10)); if the feasible region is 
bounded, then (3.8) does not hold; hence, it is known from b p > O that stepsize a 
defined by (3.11) is positive. Thus, for any value of x, in [0, a], a feasible solution 
can be determined by the equalities of (3.9), which corresponds to optimal value f/f. 
Therefore, there are an infinite number of optimal solutions. It is clear that entering 
Xq to and dropping xj, from basis give a different optimal basic solution. oO 


3.2 Getting Started 


Algorithm 3.1.1 must start from a feasible simplex tableau. In Example 3.1.1, there 
is a feasible simplex tableau available, which is not the case in general. LP problems 
are usually solved by some so-called two-phase simplex algorithm. At first, a Phase- 
I procedure is carried out to achieve feasibility, and then Algorithm 3.1.1 is used as 
Phase-II to achieve optimality. In this section, a classical Phase-I using artificial 
variables will be presented, followed by another Phase-I, termed big M method. 

To begin with, assume that all entries of the right-hand side are nonnegative, i.e., 


b >O0,i=1,...,m. 
If it is not the case, multiply by —1 each equation with a negative right-hand side. 
Then construct an auxiliary problem as follows. 


For eachi = 1, ...,m, introduce a nonnegative artificial variable xy; to the ith 
equation, and take the sum of all artificial variables as objective function, 1.e., 


m 
/ 
f= Dnt 
i=1 
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Then, using the constraint system, we eliminate all artificial variables in the 
objective, resulting in the following auxiliary problem: 


min f’ = oye bi — Qcjey aitx1 — +++ — Oy Gin) Xn 
s.t. Q11X1 +.412xX2 +++ + AinXn + Xn41 =b, 
a21X1 + a22X2 + +++ + A2nXn + Xn+2 =b 
(3.17) 
Am1X1 + Am2X2 + +++ + AmnXn + Xn4+m = bin 


xj 20, jo=Hl,...,.n+m, 


which is associated with a feasible simplex tableau, corresponding to feasible basic 
solution 


x’ =(0,...,0,b1,...,bm)! > 0. 
eee 


n m 


Therefore, Algorithm 3.1.1 can get itself started with the auxiliary problem. 
Regarding the outcome, we have the following. 


Theorem 3.2.1 The auxiliary problem (3.17) has an optimal solution. There are 
the following two cases: 


(i) If the optimal value is strictly greater than zero, the original problem is 
infeasible. 

(ii) If the optimal value is equal to zero, the first n components of the optimal 
solution form a feasible solution to the original problem. 


Proof Note that the feasible region of the auxiliary problem is nonempty, and the 
feasible objective value is bounded below by zero. According to Proposition 1.5.3, 
the auxiliary problem has an optimal basic solution. 


(i) In case when the optimal value is strictly greater than zero, the original problem 
must be infeasible, since if there was a feasible solution, say x; > 0, jf = 
1,...,7, then 


X1, +605 Xn} Xnp1 =*** =Xnim =O 


forms a feasible solution to the auxiliary problem, which is associated with zero 
auxiliary objective value. This means that there is an auxiliary objective value 
that is feasible, and less than the optimal value, which is a contradiction. 

(ii) In case when the optimal value is zero, artificial components of the optimal 
solution are all zero. Then substituting the optimal solution to the auxiliary 
constraints exhibits that the solution resulting from dropping the auxiliary 
components is feasible to the original problem. 
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Corollary 3.2.2. The original problem is feasible if and only if the optimal value of 
the auxiliary problem vanishes. 


Note that if a constraint matrix includes some columns of the unit matrix, such 
columns should be employed to reduce the number of artificial variables. The 
preceding discussions are still valid, though the auxiliary objective function involves 
artificial variables only. 

Once a feasible solution to the original problem is obtained, a feasible simplex 
tableau can be yielded by steps based on the relationship between the auxiliary 
problem and the original problem. 

Following-up steps: 


(A) Delete columns, associated to all nonbasic artificial variables (which can be 
deleted once the corresponding artificial variable leaves the basis). 

(B) Go to step D if there is no artificial basic variable. 

(C) Delete the row, associated with an artificial basic variable, if all its nonbasic 
components are zero (see the Note below); otherwise, take a nonzero compo- 
nent of it as the pivot to let the artificial variable become nonbasic, and then 
delete the associated column. This is repeated until no artificial variable is 
basic. 

(D) Overwrite the auxiliary objective row by the original costs, and then eliminate 
all basic components of this row, giving a feasible simplex tableau to the 
original problem. 


Note In step C, the row is deleted because substituting 0 to the associated 
artificial variable turns the corresponding equation into an identity, as it reflects the 
dependence of the original constraint equations. So, the method can get rid of such 
dependency. 

Overall steps can be put into the following algorithm. 


Algorithm 3.2.1 (Phase 1: Artificial Variable) This algorithm finds a feasible 
tableau: 


1. Introduce artificial variables, and construct auxiliary problem of form (3.17). 

2. Call the simplex Algorithm 3.1.1. 

3. If the optimal value of the auxiliary problem is zero, create a feasible tableau via 
“Following-up steps.” 

4. The original problem is infeasible if the optimal value of the auxiliary problem 
is strictly greater than zero. 


Example 3.2.1 Find a feasible simplex tableau to the following problem: 


min f = —x, + x2 — 2x3 

s.t. xy — 3x2 — 2x3 4+ x4 =-4 
xy —x2 + 4x3 — x5 =2 
—3x1 + x2 + x3 + 2x6 = 


xj 20, f=1,...,6. 
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Answer Construct auxiliary problem: the first constraint equation is multiplied 
by —1 to turn its right-hand side to nonnegative; as the coefficients of x¢ give a unit 


vector (0, 0, 0, 1)7, only two artificial variables x7, xg are introduced. 


min f’ = x7 + xg 


S.t. —x1 + 3x2 + 2x3 — x4 +x7 =4 
xj —x2 + 4x3 — X5 +xg=2 
—3x1 +x2 + x3 + x6 =8 


xj 20, f=1,...,8. 


Put the preceding auxiliary problem into the following tableau: 


nal x2 x3 X4 X5 X6 X7 XB ] RHS 
a Be I 4 
1 -1 4 -1 1 2 
—3 1 1 1 8 
1 1 


Turn the preceding to a simplex tableau: eliminate nonzeros in x7 and xg columns 
at the bottom (objective) row by adding —1 times of row | and of row 2 to that row: 


xX] x2 x3 x4 X5 XX %X7 Xe RHS 
-1 3 2 -1 1 4 
1 —l 4* -1 1 2 
—3 1 1 1 8 
—2 -6 1 1 | 6 


which is a feasible simplex tableau to the auxiliary problem. Call Algorithm 3.1.1 
to solve it: 
Iteration 1: 


. min{O, —2, —6, 1, 1} = -—6 <0, gq =3. 

.1={1,2, 3} 49. 

. min{4/2, 2/4, 8/1} = 1/2, p =2. 

. Multiply row 2 by 1/4, and then add —2, —1, 6 times of row 2 to rows 1,3,4, 
respectively. 


Nn BW 
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(Erase xg column after artificial variable xx becomes nonbasic): 


M23 ka 5X6 _at7_|_RHS 
3/2 7/2" =) 1/72 om 
1/4 1/41 1/4 | 1/2 
13/4 5/4 41 15/2 
a2 772 i i2 = 


Iteration 2: 


. min{3/2, —7/2, 1, -1/2} = —7/2 <0, q =2. 

-1={1, 334 9B. 

. min{3/(7/2), (15/2)/(5/4)} = 6/7, p = 1. 

. Multiply row 1 by 2/7, and then add 1/4; —5/4; 7/2 times of row 1 to rows 
2, 3, 4, respectively. 


On BR We 


(Erase x7 column after artificial variable x7 becomes nonbasic): 

Now, all the artificial variables become nonbasic; hence, reached is the optimal 
objective value 0 of the auxiliary problem. 

Overwriting the bottom row by original costs leads to 


xX] x2 4X3 x4 x5 x6 | RHS 
—3/7 1 —2/7 1/7 6/7 
1/7 1 -1/14 -—3/14 5/7 
—19/7 5/14 1/14 1 45/7 
-1 1 —2 


Adding —1 times of row | and 2 times of row 2 to the bottom row gives a feasible 
tableau of the original problem, i.e., 


Xx] X2 X83 X4 x5 x6 | RHS: 
—3/7 1 —2/7 1/7 6/7 
1/7 1 -1/14 —-3/14 5/7 
—19/7 5/14 1/14 1 45/7 
ay if =a aye 


Thus, the preceding can be taken as an initial feasible tableau to get Algo- 
rithm 3.1.1 started to solve the original problem. Solving LP problems usually 
requires two phases, both of which carry out the simplex algorithm. 

On the other hand, it seems to be attractive to solve LP problems in a single 
phase, which leads to the following so-called big-M method. 

The auxiliary problem shares the same constraints with (3.17), while its objective 
function is the sum of the original objective function and M times of the sum of all 
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artificial variables, i.e., 


min f’ = cyX1 + 2x2 +++ + CnXn + M(Xn41 + Xn42°°* + Xntm) 


S.t. A11X1 + aj2xX2 + +++ + A1nXn + Xn41 = b, 
1X1 + 22x72 + +++ + A2nXn + Xn42 = by 
Am1X1 + Am2X2 + +++ + GmnXn + Xn+m = bm 


xj 20, jol,...,n+m. 


Artificial variables in the objective function are eliminated using the constraint 
system. As a result, there will be a feasible simplex tableau to the auxiliary problem, 
which can be taken as an initial one to get the simplex algorithm started. 

The reason for using such an auxiliary objective function is as follows. Its 
artificial variable part may be regarded as a penalty function, in which the M, 
serving as penalty factor, is a sufficiently large positive number (far larger than the 
absolute value of any number involved in the computation). Such a big M imposes 
a penalty on the possible increase of values of artificial variables, consequently 
forcing artificial variables minimized before the original objective. 

It is difficult however to determine a suitable M in advance. A too large M 
could lead to bad numerical stability, while a too small M degrades the method’s 
effectiveness. It depends not only on the problem to be solved but also on the 
computer used. A practical way is to take the M as a parameter in the solution 
process. 

To demonstrate, we again bring up Example 3.2.1. Its auxiliary problem is of the 
following form: 


xX] x2 x3 X4 x5 6 2X7 2X8 RHS 
= -3.- 2 = i. | a 
1 -1 4 -1 1 2 
—3 1 1 1 8 

-1 1 —2 M M 


x] x2 X3 x4 x5 XxX6 x7 Xg | RHS 
= 3 2 = 1 | 4 
1 = 4% = 1] 2 
a 1 1 1 8 
1 1-2M -—2-6M M ™M —6M 


Thereby, the simplex algorithm can get itself stated from the preceding tableau. 
In selection of pivot column index qg, however, it should be noted that the M is so 
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large that the signs of the reduced costs depend upon coefficients of the M only. 
In the preceding tableau, e.g., x3 column is selected because M’s coefficients are 
0, —2, —6, 1, 1 in costs of nonbasic variables x; through xs, respectively, and 


min{0, =2, —6, 1, 1} = —6, q= 3. 
Row 2 is selected as pivot row by the minimum-ratio test below: 
min{4/2, 2/4, 8/1} = 1//2, p=2. 


Then elementary transformations are performed to make the associated basis 
change, completing the first iteration. 

If the process continued, it can be found that sequences of iterates created by 
the Big M method and the two-phase simplex method are the same. This is not 
surprising because the big penalty factor M forces values of artificial variables to 
vanish first before pursuing optimality of the original problem—the two methods 
are essentially the same. 

Practically, however, the two-phase method is certainly preferable to the big M 
method since the former involves no parameter and is easier to realize. In principle, 
it can be used to solve any LP problem, achieving an optimal basic solution, if any, 
or detecting infeasibility or unboundedness, otherwise. 


3.3 Simplex Algorithm 


The simplex tableau 3.1 or (3.6) is expressed at the scalar level. It is possible to 
derive a so-called revised simplex tableau at the vector/matrix level instead. 
Premultiplying the system Ax = b by B~! with transposition of terms gives 
xp +B7'Nxy = Bo'b. 
Substituting the preceding into 
ChXB +clxn —f=0 
leads to 


(ch = ch B-'N)xn == =cl Rb, 
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Table 3.2 Equivalence between the associated quantities 


Quantity Simplex tableau Relation Revised simplex tableau 
Objective row zh = cl = chB-IN 

Pivot column aq = B lag 

Right-hand side b = Bo!b 

Pivot row é, N = é, BIN 


which can be put into 


XB XN f RHS 
I B-'N Bo'b (3.18) 
cy —chB IN —1 —chB'b 


corresponding to basic solution xg = B~!b, xy = 0. The preceding, representing 
a problem equivalent to the original, is called revised simplex tableau. 

For simplicity, ah and f columns in the preceding two tableaus may be omitted, 
as they remain unchanged as the basis changes. 


Proposition 3.3.1 Any simplex tableau and revised simplex tableau, corresponding 
to the same basis, are equivalent. 


Proof Denote by (3.6) and (3.18) the two tableaus, having the same basis B. Since 
problems represented by them are equivalent, the corresponding entries of the two 
tableaus are equal. oO 


Based on the preceding Proposition, Table 3.2 gives equivalence between 
quantities of tableaus (3.6) and (3.18), involved in simplex steps. 

In simplex context, each iteration corresponds to a basis B (or the inverse B~'), 
with which entries in a simplex tableau can be calculated from the original data 
(A, b,c). Thereby, Table 3.2 will be used as a tool to derive simplex variants. 

It is noted that although the simplex tableau is calculated by Algorithm 3.1.1, 
only a part of its (m+ 1) x (7 + 1) entries are useful, that is, the objective row, 
the pivot column, and right-hand side, which are used for determining pivot. Using 
B™!, therefore, a variant can be derived by calculating the first 3 rows in Table 3.2. 

Let us consider updating of B~!. Assume that column index g and row index p 
are already determined. Replacing the pth column of B with nonbasic column ag 
gives the new basis below: 


B=(aj,..., jn 11 4qsAjngis +++ jm) (3.19) 
Then, compute B™' as follows. 


Note that ag = Boas Taking dpq as pivot, the associated elementary 
transformations amount to premultiplying the first m rows of the tableau by the 
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following m x m elementary matrix: 
1 —a1q/Apq 


—4p-1,q/4pq 
Ey= 1/apq D; (3.20) 


—4p+1,q/4pq 
—Amq/Apq 1 


P 


which may also be obtained by executing the same elementary transformations on 
the unit matrix. Such a matrix, which is the same as the unit matrix except for the 
pth column, is determined only by ag. Combining (3.19) and (3.20) gives 


= -1,. -1,. -1 -1,. Shes 
E,BB=E,(Baj,,...,B aj, ,B aq, B Gj geese B jn) 
= Ep @i,--+,€p—1s qs Cpt, +++ em) 
= (Epei,..., Epep-1, Epdg, Epep+i,.-., Epem) = 1, 
from which the recursion formula follows, i.e., 
B1—z,Ap (3.21) 
=Ep : : 


Based on the preceding discussions and the equivalence between the simplex 
tableau and revised simplex tableau, we can turn Algorithm 3.1.1 into the following 
revision (Dantzig and Orchard-Hays 1953): 


Algorithm 3.3. 1 (Simplex Algorithm 1) Initial: (B, N), B~',xg = B~'b > 0, 
and f = clap. This algorithm solves the standard LP problem (. 8). 


= 


Compute Zy = cn — Nt y y= Bo" cp. 

Determine pivot column index q € argmiNjen Z;. 

Stop if Zq = 0 (optimality achieved). 

Compute ag = Bde. 

Stop if dg < 0 (unbounded problem). 

Determine stepsize a and pivot row index p such that: 
a = Xj,/Apg = min{xX;,/dig | dig > 0; i=1,...,m}. 
7. SetXq = a, and update xg = XB — ag, f= Fae: ifa #0. 
8. Update B~' by (3.21). 

9. Update (B, N) by exchanging jp and q. 

0. Go to step 1. 


AwMARWN 
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In step 1, vector y is calculated first, and it is then used to compute the reduced 
costs zy, which is referred to as pricing. y is called simplex multipliers (vector), 
whose additional meanings will be clear later. 


Example 3.3.1 Solve the following problem by Algorithm 3.3.1: 


min f = —4x1 — 3x2 — 5x3 
s.t. 2x, + x2 + 3x3 + x5 = 15 
Xj +x2+%34+%4 = 12 
x2 — 3x3 +2x7=3 
2x1 + x2 + x6 =9 


xj>0, j=l,...,7. 


Answer Initial: B = {5,4,7,6},N = {1,2,3},B°! = JIxgp = 
(15, 12,3,9)', f =0. 
Iteration 1: 


1. § = BT cg = (0,0, 0,0)", Zy =ew — N75 = (—4, -3, —5). 
2. min{f—4, —3, -5} = —5 <0, g =3. 
4. a3 = Bo'a3 = (3, 1, —3, 0)". 
6. a = min{15/3, 12/1} = 15/3 =5, p=1. 
7. xp = (15, 12,3,9)' — 5 x (3,1, —3,0)' = (0,7, 18,9)", 13 = 5, f = 5x 
(—5) = —25. 
1/3 
_,_ | -1/31 
8. B= : 3 
0 1 


9. B= {3,4,7, 6}, N = {1, 2,5}, xp = (5,7, 18,9). 
Iteration 2: 


1.7 = (—5/3,0,0,0)', zy = (—4,—3,0)' — (—10/3, —5/3, —5/3)7 
(—2/3, —4/3, 5/3)". 


2. min{—2/3, —4/3, 5/3} = —4/3 <0, q =2. 
4. a = (1/3, 2/3, 2, 1)". 
6. @ =min{15, 21/2, 9,9} =9, p =3. 
7. Xp = (5,7, 18, 9)? —9 x (1/3, 2/3, 2, 1)? = (2, 1,0, 0)', x2 = 9, f =—25+4 
9 x (—4/3) = —37. 
1 -1/6 1/3 1/6 —1/6 
gga. fat) —1/31 _ | =2/3 1-1/3 
0 1/2 1 1 1/2 1/2 
0 -1/21 0 1 =172) =1/2 1 


9. B = {3,4, 2,6}, N = {1,7,5}, xe = (2, 1,9, 0)!. 
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Iteration 3: 


1. ¥ = (-7/3,0, —2/3,0)', zy = (—4,0,0)' — (—10/3, —2/3,-7/3)' = 
(—2/3, 2/3, 7/3)'. 


2. min{—2/3, 2/3, 7/3} = —2/3, q = 1. 

4. a = (2/3, 1/3, 0, 2)!. 

6. a = min{2/(2/3), 1/(1/3), 0/2} =0, p = 4. 

7. Xp = (2,1,9,0)', x6 =0, f = —37. 
1 -1/3 1/6 —1/6 1/3 0 -1/3 

g pata | Ol -V6][-231-18 |_| -7/121-14-1/6 
0 1 0 1/2 1/2 1/2 1/2 0 
0 1/2 —1/72 -1/21 —1/4 -1/4 1/2 

8. B = {3,4,2, 1}, N = {6, 7, 5}, xB = (2, 1,9, 0)!. 

9. Omitted. 


Iteration 4: 


1. y = (—13/6, 0, —1/2, —1/3)', zy = (0,0, 0)? — (—1/3, —1/2, -13/6)? = 
(1/3, 1/2, 13/6)" = 0, 
2. The optimal basic solution and optimal value: 


x* = (0,9, 2, 1,0,0,0)', jf? = 37, 


It is possible to derive a variant, in which row p is computed instead of column gq, 
and objective row is computed by recursion (see (3.14)). 


Algorithm 3.3.2 (Simplex Algorithm 2) Initial: (B, N), B~', x3 = B~'b > 0, 
ZN = Cn — N™B-' cp, and f = Chae This algorithm solves the standard LP 
problem (1.8). 


1. Determine pivot column index q € arg min jen Z;. 
2. Stop if Zq = 0 (optimality achieved). 
3. Compute ag = Bo dy. 
4. Stop if dg < 0 (unbounded). 
5. Determine stepsize a and pivot row index p such that 
a = Xj,/Apg = Min{xX;,/dig | dig > 0; i=1,...,m}. 
6. Set Xq = a, and update xg = XB — ag, f=fr+ aZq ifa £0. 
7. Compute oy = N'v, where v = Bole, 
8. Update by: zy = zn + Bon, Zj, = B, where B = —Zq/apq.- 
9. Update B™' by (3.21). 
0. Update (B, N) by exchanging jp and q. 
1. Goto step 1. 


Algorithm 3.3.1 was derived based on the equivalence of the simplex tableau and 


revised simplex tableau. Alternatively, it can be derived by calculating a descent 
edge, emanating from the current vertex, as the search direction. 
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Without loss of generality, let B = {1,...,m} and N = {m+1,...,n} 
be, respectively, the basis and nonbasis, associated with feasible basic solution x. 
Assume that pivot column index qg € N has been determined such that 


Introduce the vector 


-1 
APS (es oa ) (3.22) 


—€q—m 


where €,— is the (1 — m)-dimensional unit vector with the (¢ — m)th component 
1. It is clear that 


—cTAx = ¢q—ajB cy =Z%q <0. (3.23) 


Therefore, —Ax is descent with respect to c'x. Taking it as search direction gives 
the following line search scheme: 


=x -—aAx, (3.24) 


where a > 0 is a stepsize to be determined. 
Since x is feasible, it holds for any a > 0 that 


Ak = Ax —a[B, NJAx = Ax =b. 


Therefore, maximizing w subject to tg > 0 is reasonable. When B lag £ 0, such 
doing results in a and p such that 


=F; )(B? apg) = mini, (Bldg) |B ag = 0,4 = 12.5): 
(3.25) 


It is clear that the associated new solution < is still feasible. In fact, it is verified that 
x is just feasible basic solution, associated with the new basis, resulting from the 
old by exchanging jp and q. 

The relation between the new and old basis matrices is 


B=B + (aq — aj, )es,- 


In view of that a;, is the pth column of B and that Bola; = ép and Boag = 
Gq hold, it is not difficult to derive the following result from Sherman—Morrison 
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formula (Golub and Van Loan 1989): 
a Bo (aq —- a;,Je,B oe (Gq — ep)ey 
“pq 


Roi ape ae 3.26 
le, B 1 (ag —a;,) a 


which may serve as an update of B™!. In fact, it is verified that the preceding 
and (3.21) are actually equivalent. 
The search direction —Ax, defined by (3.22), can be further investigated 
geometrically. Concerning the set 
E={xeR" | Ax=b,x>0, xj =0,ga4j EN} 
= {xe R" | xp = B'b—x,(B ag) = 0, xg 20, x} =0,g J EN}, 
(3.27) 


we have the following result. 


Proposition 3.3.2 The set E is a descent edge, emanating from the current vertex 
x, and —Ax is its direction. Ra, < 0, then —Ax is an (unbounded) extreme 
direction. 


Proof It is clear that E is a half-line or edge, emanating from x. By (3.27), (3.25), 
and (3.22), for any x € E C P, it holds that 


E={xeR" | xp = B'b—xq(B"'aq), OX xq <a, xj =0,9 4 EN} 


={xeR” | x =x —xgAx, Xq € [0, a)}. 
By (3.23), it is known that the associated objective value satisfies 
f= clx =clx — age Re =clk+ XgZq S om, 


Note that (3.25) is well-defined when Ba, > 0. If, in addition, x;, = 0, then 
a = 0, and hence E degenerates to vertex x. If Xj, > 0, hence a > O, then the 
associated objective value strictly decreases with x, € [0, a]. Therefore, —Ax is 
a direction of the descent edge E. When Boag < 0, it is clear that a = +00 
corresponds to the edge E € P, and hence, — Ax is an extreme direction. oO 


3.4 Degeneracy and Cycling 


It was seen that a zero stepsize leads to the same feasible basic solution. Therefore, 
the finiteness of the simple method is questionable. Soon after its emerging, it is 
found that the simplex method does not terminate in a few cases. E.M.L. Beale 
(1955) and A.J. Hoffman (1953) offered such instances independently. 
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The following is due to Beale. 


Example 3.4.1 Solve the following problem by Algorithm 3.1.1: 


min f = —3/4x4 + 20x5 — 1/2x6 + 6x7 


s.t. x1 + 1/4x4 — 8x5 — x6 + 9x7 =0 
x2 + 1/2x4 — 12x5 — 1/2x6 + 3x7 = 0 
x3 + X6 =1 


MEO, GS TpcengT 


Answer Initial: The following feasible tableau is available from the preceding: 


Xp X22 x3 x4 x5 X6 RHS 
1 1/4* -8 -1 
1 1/2 —-12 —-1/2 
1 1 1 


Iteration 1: 


. min{—3/4, 20, 1, —1/2, 6} = —3/4 <0, g=4. 

.1={1, 23490. 

. min{0/(1/4), 0/(1/2)} =0, p= 1. 

. Multiply row 1 by 4, and then add —1/2,3/4 times of row | to rows 2,4, 
respectively: 


mn BR We 


x1 XQ X30 X4 X5 X6 RHS 
4 1 —32 —4 
—2 #1 4* 3/2 
1 1 1 
3 -4 —-7/2 


Iteration 2: 
1. min{3, —4, —7/2, 33} = —4 <0, g=S. 


3. 1=Q 49. 
4. min{0/4)} =0, p =2. 
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5. Multiply row 2 by 1/4, and then add 32,4 times of row 2 to rows 1,4, 


respectively: 


Xx] x2 X3 X4 X5 X6 
—12 8 1 8* 
-1/2 1/4 1 3/8 

1 1 
1 1 a2, 


Iteration 3: 


1. min{1, 1, —2, 18} = —2 <0, gq =6. 
3.1 ={1,2,3} 49%. 
4. min{0/8, 0/(3/8), 1/1} =0, p=1. 
5 
respectively: 
x1 x2 X3 x4 x5 
—3/2 1 1/8 
1/16 —-1/8 —3/64 1 
3/2 -1 1 —1/8 
=) 3 1/4 


Iteration 4: 

. min{—2, 3, 1/4, —3} = —-3 <0, q =7. 
.1= {2,3} 40. 

. min{0/(3/16), 1/(21/2)} =0, p =2. 


an BW ee 


1, 3, 4, respectively: 


XxX] x2 X3 X4 x5 
2* 6 5/2 56 
1/3 278 1/4 16/3 
=) ae 1 5/2. —56 
—-1 1 —1/2 16 


Iteration 5: 
1. min{—1, 1, -1/2, 16} = —-1 <0, q=1. 


3.1 = {1,2} 49. 
4. min{0/2, 0/(1/3)} =0, p =1. 


RHS 


RHS 


. Multiply row | by 1/8, and then add —3/8, —1, 2 times of row | to rows 2, 3, 4, 


RHS 


. Multiply row 2 by 16/3, and then add 21/2, —21/2, 3 times of row 2 to rows 
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5. Multiply row | by 1/2, and then add —1/3, 2, 1 times of row | to rows 2,3,4, 
respectively: 


xi x2 x3 x4 x5 x6 x7 RHS 
i 3 5/4 28 1/2 
1/3* 1/6 -4 -1/6 1 
1 1 1 
=) -7/4. 44 1/2 


Iteration 6: 


1. min{—2, —7/4, 44, 1/2} = —2 <0, g=2. 

3.1 = {2} 490. 

4. min{0/(1/3)} = 0, p = 2. 

5. Multiply row 2 by 3, and then add 3, 2 times of row 2 to rows 1, 4, respectively: 


XxX x2 x3 X4 x5 X6 x7 RHS 
1 1/4 —8 -1 
1 1/2 -12 -1/2 3 
1 1 1 


-3/4 20 —-I/2 6 


It is seen that stepsizes are equally zero in all the 6 iterations, and the last tableau 
is the same as the first. Thus, continuing the process always generates the same 
sequence of tableaus, which is called cycling. Consequently, the simplex algorithm 
failed to solve Beale’s problem. It is clear that cycling could occur only in case of 
degeneracy. 

As was addressed in Sect. 2.8, the simplex method can be explained in terms of 
the active set method. In each iteration, a vertex is actually determined by Ax = b 
and x; = 0, j € N corresponding to n active constraints. Since it has zero basic 
components, a degenerate vertex is the intersection of hyperplanes, the number of 
which is strictly greater than n. At first glance, this situation seems rarely to occur 
in practice. Surprisingly, however, it is just the opposite: problems from practice are 
almost all degenerate. 

The real problem caused by degeneracy seems to be stalling, which degrades the 
method’s performance seriously since a large number of iterations stay at a vertex 
for too long a time before exiting it. It is especially a headache for highly degenerate 
problems, in which the vanished basic components of the solution occupy a very 
high proportion, leading to a huge number of iterations required. 

As a preliminary countermeasure, we put forward the following would be 
favorable variant of the classical row rule. Some newly proposed pivot rules will 
be presented in Part II. 
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Rule 3.4.1 (Row Rule) Define I = {i | dig > 0, i=1,...,m}, hh = {i | bi = 
0, i € I}. Determine pivot row index p and stepsize a by 
, arg max{d; q | ie€ hi}, a= 0, ; if 1; Oo, (3.28) 
argmin{bj/dig | i € I}, a= bp/dpq, otherwise. 


3.5 Finite Pivot Rule 


As was shown in the previous section, the finiteness of the simplex method is not 
guaranteed. An approach or pivot rule that turns the method to finite one is called 
finite. 

Is there any finite approach/rule? The answer is positive. This is the theme of this 
section. 

For this purpose, Charnes (1952) added a perturbation term to the right-hand 
side, that is, 


2 a ae 


where € > 0 is a sufficiently small parameter while still using classical pivot rule. 
Theorem 3.5.1 Charnes’ perturbation approach is finite. 


Proof Put the perturbation term in the form w = /w so that in any iteration, the 
right-hand side can be written as 
v=b+Uw, 

where U is a permutation matrix, resulting from performing related elementary 
transformations on J. Note that U and J have the same rank m, and hence, each row 
of U is nonzero. First, it holds that b > 0, because if, otherwise, b; < 0 for some 
i € {1,..., m}, then it follows that v; < 0, as it contradicts to feasibility. Further, it 
is clear that v; > O holds for all row indices i, satisfying b; > 0; on the other hand, 
v; > 0 also holds for all row index i with b; = 0, since the first nonzero of the ith 
row of U is positive (otherwise, it contradicts feasibility). Therefore, v > 0 holds. 
Since each tableau corresponds to a nondegenerate feasible basic solution, there 
is no possibility of cycling; hence, the process terminates within a finite number 
of iterations. Consequently, eliminating all parametric terms in the final optimal 
tableau gives an optimal one to the original problem. oO 


Once a pivot column index q is determined, Charnes’ approach amounts to 


selecting pivot row index p such that 


p € argmin{(b; + uj1€ ture? +++» + uime™)/Gig | Gig > 0; i=1,...,m}. 
(3.29) 
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Dantzig et al. (1955) proposed a pivot rule based on the so-called lexicographic 
order. This order compares vectors by their first different components. Normal equal 
vectors are also regarded as equal. (Aj,..., Ay) < (M4,..., pa) means that the 
former is less than the latter in lexicographic order, that is, for the smallest subscript 
i, satisfying 4; A j4;, it holds that A; < j;. Similarly, the “>” denotes “greater 
than” in lexicographic order. 


Rule 3.5.1 (Lexicographic Rule) Given pivot column index q, the row index p is 
determined such that 


p € argmin {(b;, uj 1, Ui2,---,Uim)/dig | Gig > 0; i=1,...,m), (3.30) 


where u; ; denotes the entry of U, and “min” means minimizing the objective 
function in the sense of lexicographic order. 


The preceding is equivalent to (3.29), in which € is sufficiently small. Therefore, 
it is a finite rule. 

Among the existing finite rules, Bland rule (1977) draws considerable attention 
due to its simplicity. 


Rule 3.5.2 (Bland Rule) Among nonbasic variables, corresponding to the negative 
reduced costs, select the smallest-indexed one to enter the basis. When there are 
multiple rows, attaining the same minimum ratio, select the basic variable with the 
smallest index to leave the basis. 


Theorem 3.5.2. The Bland rule is finite. 


Proof Assume that cycling occurs with the simplex algorithm using Bland rule. If 
some variable leaves the basis in a circle, it must enter the basis gain. Denote by T 
the index set of these shuttling variables, and define 


t=max j €T. 


Note that the stepsize is always equal to 0 in each iteration in the circle and hence 
leads to the same feasible basic solution; besides, the h-indexed component of the 
feasible basic solution is 0 for any h € T. 

Assume that x; is selected to enter the basis for simplex tableau 


= , 3.31 
zé -1 | B een) 
thus z; < 0, and z; > O for any reduced cost’s index j < f. 
Assume that at another simplex tableau 
A 0 b 
7m =, 3.32 
i B ( ) 
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basic variable x; in row p leaves and nonbasic variable x; enters the basis. Let 
Xj,1+++sXj, be basic variables (x;, = x;). It follows that ¢; < 0, and ¢; > 0 for 
any reduced cost’s index j < s. Note that the pivot is positive, ie., dp; > 0; since 
s € T, therefore, it holds that s < tf. 


Define vz, k = 1,...,n,n +1 as follows: 
1, k=s, 
Sty kai, Tal. casm, 
=) 4 3.33 
a Cs, k=nt+l, ( ) 
0, otherwise. 


Note that basic columns of A constitute a permutation. Nonbasic components of the 
vector 


T 
v= (vj,..., Un, Un+1) 
are all 0, except for v; = 1. Fori = 1,...,m, on the other hand, the basic entries 
in row i of A, except for Gi j; = 1, are zero; basic entries of ¢ are zero as well. 
Therefore, it holds that 
a ay Ss 
A 0 
& I v=]° . = 0. (3.34) 
c — Ams — Ams 


Cs — Cs 


Since (3.31) can be obtained from (3.32) by premultiplying a series of elementary 
matrices, it follows that 


(Fr, )¥=9, (3.35) 


where the last equality is 


n 
Y> Zeve — Ung = 0, 
k=1 


or 


= 


ZkUk = Un+1 = Cs < 0. 


> 
ll 
a 
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Therefore, there exists some index h < n + | such that 
Zhup < O, (3.36) 


giving 7, 4 Oand vw, 4 0. 
On the other hand, it is known by vy, ¥ O and the definition of v that h € 
{j1,---, jm, 5}. Thus, there are only the following three cases: 


G) h = s. vy = 1 in this case. Since x; is an entering variable for simplex 
tableau (3.31) andh = s < t, hence z, > 0, it follows that z,v, = Z, > 0, 
contradicting (3.36). 

(ii) h = jp =t. In this case, from Z, = Z; < O and vq = —Gps < 0, it follows that 
ZnvUpn > O, contradicting (3.36). 

(it) h = jj # jy orh A t. Now xp is a nonbasic variable of simplex tableau (3.31) 
(otherwise, Z, = 0); it is also a basic index of simplex tableau (3.32), thus 
h € T. It follows that 


b; = 0, h<t, (3.37) 
and hence z;, > O. Further, it holds that 
UV, = —dij.s > 0, 


since, otherwise, v,» #~ 0 gives aj; > 0, from which and (3.37) it follows 
that x, rather than x;, were selected to leave the basis for tableau (3.32), as a 
contradiction to (3.36). Therefore, Bland rule is finite. oO 


Terlaky (1985), and Wang (1987) independently proposed a criss-cross finite 
variant of Bland rule, which is embedded in a somehow different context, compared 
with the simplex method. 

Unfortunately, it turns out that these finite rules are very slow in practice and 
cannot be mentioned in the same breath as the classical rule. This is not surprising, 
however. For example, Rule 3.5.2 gives nonbasic variables with small index priority 
to enter the basis, while we all know that basic variables of an optimal solution are 
not necessarily small indexed. 

Rule 3.5.2 actually uses a priority order, coinciding with decreasing indices, for 
the selection of the entering variable. It is clear that the “ideal” order, if any, should 
enter a basic variable of an optimal solution to basis. According to the heuristic 
Proposition 11.1.2, inequality constraints with small pivoting-indices should be 
satisfied as equations, or the corresponding variables would be better to be nonbasic. 
Therefore, variables with larger pivoting-indices have priority to enter the basis. 

Thus, we have the following variant of Bland rule (Pan 1990). 


Rule 3.5.3. Among nonbasic variables, corresponding to the negative reduced 
costs, select the maximum pivoting-indexed to enter the basis. When there are 
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multiple rows, attaining the same minimum ratio, select the maximum pivoting- 
indexed basic variable to leave basis. When there is a tie, take the maximum indexed. 


Theorem 3.5.3 Rule 3.5.3 is finite. 


Proof This rule is equivalent to Rule 3.5.2 if variables are re-given indices in 
accordance with their pivoting-indices. Oo 


Preliminary computational experiments with small problems show that perfor- 
mance of Rule 3.5.3 is much better than Bland Rule. It might be the best among 
known finite rules. However, it is still inferior to the conventional rule, requiring 
more iterations in general. 

The Bland Rule can be generalized to the following finite rule. 


Rule 3.5.4 Assume we are given an order for the variables. Among nonbasic 
variables, corresponding to the negative reduced costs, select one the smallest in this 
order to enter the basis. When there are multiple rows attaining the same minimum 
ratio, select the basic variable smallest in the order to leave basis. 


In Example 3.4.1, we have seen that cycling occurred with the simplex algorithm 
with the conventional rule. The situation is different if Rule 3.5.3 instead. 


Example 3.5.1 Solve the Beale problem by Algorithm 3.1.1 using Rule 3.5.3: 


min f = —3/4x4 + 20x5 — 1/2x6 + 6x7 


s.t. x] + 1/4x4 — 8x5 — x6 + 9x7 =0 
x2 + 1/2x4 — 12x5 — 1/2x6 + 3x7 =0 
x3 + x6 =1 


xj>0, j=l,...,7. 


Answer As the coefficient matrix includes a unit matrix, it is easy to transform 
the constraints to the “>” type of inequalities: 


min f = —3/4x4 + 20x5 — 1/2x6 + 6x7 
S.t. —1/4x4 + 8x5 + x6 — 9x7 
—1/2x4 + 12x5+ 1/2x6 — 3x7 
-X6 


IV IV IV 


Note that the first 3 constraints correspond to the original variables x1, x2, x3, 
respectively; pivoting-indices of constraints may be regarded as those for the 
associated variables. 


It is noted that the gradient of the objective function is c = (—3/4, 20, —1/2, 6)". 
The gradient of the first constraint is aj = (—1/4,8,1, —9}!, and the calcu- 
lated pivoting-index of this constraint (or corresponding variable x;) is aj = 
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—a}c/lla || = —8.74. Similarly, we calculate all pivoting-indices and put them 
into the following table in decreasing order: 


Variabe 


X4 
X6 
X3 
X7 


x1 
X5 


x2 + 12x5 


Now call Algorithm 3.1.1 with Rule 3.5.3. 
Initial: The following feasible simplex tableau is obtained directly from the preced- 
ing problem: 


Xp X2 X3 x4 x5 x6 RHS 
1 1/4* -8 -1 
1 1/2 —12 —-1/2 
1 1 1 


Iteration 1: 


1. Among nonbasic variables x4 (@4 = 0.75) and x6 (a = 0.50) corresponding 
to negative reduced costs, select the maximum pivoting-indexed x4 to enter the 


basis, g = 4. 

3.1 = {1,2} 49. 

4. min{0/(1/4), 0/(1/2)} = 0. Among basic variables in rows 1 and 2, select the 
maximum pivoting-indexed x; (a} = —8.74 > —114.78 = az) to leave the 
basis, p = 1. 


5. Multiply row | by 4, and then add —1/2,3/4 times of row | to rows 2, 4, 
respectively: 


Xx] xX. X3 X4 X5 XG RHS 
4 1 —32 —4 
—2 1 4 3/2* 
1 1 1 
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Iteration 2: 


1. Among nonbasic variables x5 (#5 = —20.00) and x6 (a@& = 0.50) with negative 
reduced costs, select the maximum pivoting-indexed x¢ to enter the basis, gq = 6. 
.1= {2,33} 490. 
. min{0/(3/2), 1/1} = 0, only x2 is eligible for leaving the basis, p = 2. 
5. Multiply row 2 by 2/3, and then add 4, —1, 7/2 times of row 2 to rows 1, 3, 4, 
respectively: 


RW 


XxX] x2 x3 X4 X5 X6 X7 RHS 
4/3 8/3 1 64/3 —4 
=e, 2/3 8/3. 1 10 
4/3 —2/3 1 —8/3 10* 1 
-5/3 7/3 16/3 2 
Iteration 3: 
1. Among nonbasic variables x; (#1 = —8.74) and x7 (#7 = —6.00 with negative 


reduced costs, select the maximum pivoting-indexed x7 to enter the basis, g = 7. 
.1L= Bh) 49. 
. Only x3 is eligible for leaving the basis, p = 3. 
5. Multiply row 3 by 1/10, and then add 4, 10, 2 times of row 3 to rows 1, 2, 4, 
respectively: 


RW 


XxX] x2 X3 X4 x5 X6 x7 RHS 

—4/5 12/5. 2/5. 1112/5 2/5 
1 1 1 

2/15* -1/15 1/10 —4/15 1 | 1/10 

-7/5 W/5 1/5 24/5 1/5 


Iteration 4: 


1. Only nonbasic variable x; is eligible for entering the basis, g = 1. 
3. 1={3} 49. 
4. Only x7 is eligible for leaving the basis, p = 3. 
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5. Multiply row 3 by 15/2, and then add 4/5; 7/5 times of row 3 to rows 1, 4, 
respectively: 


xX] x2 X3 x4 X5 X6 X7 RHS 
2 2/5 1 —24 6 1 

1 1 1 

i <1. 3/4 = 15/2 | 3/4 
3/2 5/4 2 21/2 | 5/4 


All the reduced costs are now nonnegative. The optimal solution and optimal value 
are 


x* = (3/4, 0,0, 1,0, 1,0)", f* =—5/4. 


Thus, the Beale problem is solved without cycling. 


3.6 Notes on Simplex Method 


Related to the linear structure of the LP model, the simplex method is pivotal in 
nature, which progresses through a series of pivot operations (for the origin of the 
simplex method, see Appendix A). Its iteration is characterized by a basis: once a 
basis is known, so is the corresponding feasible basic solution. It selects a pivot and 
makes the corresponding basis change to improve the objective value until reaching 
the optimality unless detecting the unboundedness. Therefore, the pivot selection is 
crucial to the simplex method. 

The past research on the simplex method has been on the two aspects: on one 
side, new pivot rules were suggested from time to time even if Dantzig’s original 
rule has been widely used in practice for many years. On the other hand, efforts 
have been made to improve its framework. 

The evaluation of an algorithm involves the number of required arithmetics and 
storage, numerical stability, and difficulty for programming. Its time complexity 
and storage complexity depend on the size of the problem handled. For the standard 
LP problem, its size may be measured by (m,n) roughly. However, the true size 
depends on values of (A, b, c), as can be determined by the total number L, called 
input length of binary digits of input data. If the number of arithmetics is bounded 
above by some function tf(m,n, L), where t > O is some constant, the time 
complexity of the algorithm is said O(f(m,n, L)). If f(m,n, L) is polynomial 
in m,n, L, itis of polynomial-time complexity, as is commonly regarded as “good.” 
If f(m,n, L) is exponential in m,n, L, it is of exponential time complexity, as 
regarded as “bad” since it can fail to solve large-scale problems by consuming 
too much time. Note, however, that this is just the worst-case complexity since 
tf (m,n, L) is an upper bound of the required arithmetics. 
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The following table lists the numbers of arithmetics per iteration and storage 
locations required by tableau simplex Algorithm 3.3.1 vs. the revised simplex 
Algorithm 3.1.1: 


Algorithm 3.3.1 Algorithm 3.1.1 

Multiplications | m(n+2m)+2m+1 |) (m+)D(in—m-+1) 
| Additions m(n + 2m) —2m +1 m(n—m +1) 
Storages (m + 1)(m + 2) (m+ 1)(n+ 1) 


In the preceding, the number of storage locations required by Algorithm 3.1.1 
excludes that for the original data (A, b, c), though these data should be stored, in 
computer. Therefore, the storage requirement of the tableau simplex algorithm is 
significantly high, compared with the revised simplex algorithm, especially when 
n>m. 

The two algorithms require the same iterations in solving any standard LP 
problem, in principle. Therefore, we only compare the arithmetics, mainly multi- 
plications, in a single iteration. The table shows that the revised simplex algorithm 
is better than the tableau simplex algorithm, especially when n >> m. For sparse 
computations, the former is even far superior. It is not surprising that the latter 
appears only in textbooks. 

The time complexity of the simplex method mainly depends on the required 
number of iterations. Klee and Minty (1972) offered an example indicating that 
the simplex method using the classical pivot rule passes through all the 2” vertices 
of the feasible region. Therefore, the rule is not polynomial-time, not to mention 
the instances in which the simplex method fails to solve because of cycling. It 
turns out that many other rules, such as the most improvement rule, the steepest- 
edge rule, even Bland’s finite rule, etc., are all not polynomial-time. It has not been 
clear whether there exists a polynomial-time rule though such possibility seems 
to be very low if any. Indeed, the simplex method is unsatisfactory for solving 
a certain type of LP problems, e.g., hard large-scale problems or problems with 
combinatorial constraints, such as those with zero/one coefficients or those from 
Krawchuk polynomials. Schrijver (1986, p.141) wrote: 


The main problem seems to be that the simplex method is “myopic,” cannot see “cut-off 
paths.” It does not find a path which is locally bad but globally good. 


Nevertheless, the evaluation of methods is essentially a practical issue. The 
simplex method performs very well and has been very successful in practice. For 
solving small/medium LP problems, in particular, it usually requires iterations no 
more than 4m to 6m (including Phase-1). Therefore, the worst-case complexity has 
a limited value and is even misleading. 

On the other hand, the complexity under some probability sense seems to be 
closer to reality. Borgwardt (1982a,b) showed that an average complexity of the 
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simplex method is polynomial. Specifically, for the LP problem 


min c'x 


s.t. Ax <b, 


where A € R”™*", b > 0, and components of c are random variables under certain 
assumptions, he proved that the mathematical expectation of iterations, required by 
the simplex method using a special pivot rule, is 


On mVO-D), 


Using another probability model and pivot rule, Smale (1983a,b) proved that the 
average complexity of the simplex method is bounded above by 


O(log n)"*"), 


which is not polynomial, but still better than Borgwardt’s result when n >> m. These 
results coincide with reality. 
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Chapter 4 ® 
Implementation of Simplex Method od 


All algorithms formulated in this book, such as the simplex algorithm and the dual 
simplex algorithm, are theoretical or conceptual and cannot be put into use directly. 
Software, resulting by the following algorithms, step by step would solve textbook 
instances only. Implementation techniques are crucial to the success of optimization 
methods. 

This chapter will highlight implementation issues for the simplex method. There 
are two aspects: one is to reduce involved computational effort and the other to 
improve numerical stability. Only limited sparse techniques will be considered, as 
a full treatment of sparsity is beyond the scope of this book. Associated contents 
include the miscellaneous, including feasibility/optimality tolerance, machine 0, 
etc., and scaling, LU factorization as well as sparse LU factorization of basis, updat- 
ing LU factors, crashing for initial basis, Harris pivot rule, tolerance expending, and 
pricing scheme of the reduced costs. 


4.1 Miscellaneous 


As rounding errors affect computational procedures and results (see Sect. 1.1), 
restricting such effects to improve the numerical stability is important to the 
implementation of the simplex algorithm. 

First, theoretical 0 or oo involved in the algorithm should be treated properly by 
giving tolerances, corresponding to the computer precision used. Based on MINOS! 
(Murtagh and Saunders 1998), for instance, main parameters are listed below, where 


' MINOS is an optimization package developed by Murtagh and Saunders at Dept of Management 
Science and Engineering of Stanford University. 
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the computer precision is assumed to be 


éa3- 22972 i0-*. 


Introduce 
€o = €93 & 3.00 x 10713, 
a= 9! = 3.25 x 10", 
featol = 10~°, 
plinfy = 107°, 
where: 


¢ Quantity in the solution process, whose absolute value is no more than €, is 
treated as machine 0. 

* Quantity, whose absolute value is more than plinfy, is treated as oo. 

* tolpiv = €; is used to avoid a too small pivot; e.g., the minimum-ratio test (3.11) 
is actually replaced by 


a= Dilng = min{b; /G; g | dig > tolpiv, i=1,...,m} >= 0. (4.1) 


Thereby, any pivot chosen satisfies dy), > tolpiv. The problem is regarded as 
unbounded if no such a pivot is found. 


The magnitude of a pivot seriously affects the numerical stability of the 
algorithm. The determination of the numerical nonzero pivot not only depends on 
the computer precision but also on the problem itself to be solved. It is difficult to 
give the “best” tolpiv. Some value between €, and 107’ seems to be acceptable, 
though still far from satisfactory. This theme will be discussed further in Sect. 4.7. 


¢ featol is a primal or dual feasibility tolerance. For j = 1,--- ,n, xj; = —featol 
is used to replace the nonnegativity condition x; > O (for primal feasibility). 
For j € N, in optimality test, z; = —featol is used to replace z; > O (for dual 
feasibility). Therefore, the column rule (3.13) is replaced by 


q € arg min{Z; | Zz; < —featol, j < N}. (4.2) 
If there is no such an g found, i.e., 
zj = —featol, Vj € N, 


then optimality is declared. 

In the solution process, however, it is not easy to ensure all solutions to be 
within the feasibility tolerance, let alone degeneracy being intertwined with this. 
Solving large-scale LP problems usually involves a large number of iterations. 
After many iterations, due to accumulated errors, intermediate results could 
be too inaccurate to be used. It is, therefore, necessary to perform a so-called 
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restarting strategy: rebooting the process from scratch, by recomputing from the 
original data, inverse B~! or its LU factorization of the current basis. It might be 
well to restart every 100 iterations. In addition, residuals of equality constraints 
at iterates should be monitored in the solution process: once residuals are found 
to exceed some tolerance, carry out the restarting strategy, it is also necessary 
to predetermine an upper bound on the number of iterations to limit running 
time. Once iterations exceed such a bound, terminate the solution process after 
recording the end basis for a possible later warm start. 

Real-world demand is a powerful pushing force for implementation tech- 
niques for solving large-scale LP problems. Nowadays, LP problems yielding 
from practice become larger and larger. Huge problems are quite common. The 
solution to such types of problems is very challenging. Storage required would 
be too large to meet; even if it is met, the running time consumed would still be 
too high to afford. 

Like many other models, fortunately, most large-scale LP problems are sparse, 
i.e., data A, b,c involve a large number of zeros. Usually, the proportion of 
nonzeros of A is no more than 1%, hardly 5%. This is so because variables, 
involved in large systems, usually belong to different groups corresponding to 
different regions, departments, or properties, and so on), and constraints mainly 
reflect relations within the interior of a group, so that each variable would appear 
in a few constraints. With the help of sparse matrix techniques, which are quite 
mature nowadays, it is possible to proceed without storing and manipulating 
zeros at all. Therefore, the key to solving large-scale sparse problems is to take 
advantage of sparsity to reduce storage and arithmetic requirements. In profes- 
sional optimization packages, the original nonzero data are usually expressed via 
a so-called MPS format in an input data file (Appendix B). 


4.2 Scaling 


The large difference within the original data affects calculated results considerably, 
which is related to the accumulation of rounding errors. To reduce such differences, 
the original data should be preprocessed by some so-called scaling technique 
(Hamming 1971). 

Adjusting the unit of measurement is a common scaling method. It amounts to 
performing variable transformation x = D,x', where D, called scaling factor, is a 
diagonal square matrix with positive diagonals. For instance, simply taking 


I/llail| 


1/llaa|| 
Di = 7 (4.3) 


1/\lan|l 
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converts the standard LP problem into 


min f = (cP D)2", (4.4) 

s.t. (AD))x'=b, x’ >0, 
where each column of A D, has norm 1, and hence, differences between columns are 
balanced. Then, once an optimal solution x’ to (4.4) is obtained, the corresponding 
optimal solution to the original is readily available, i.e., x = D, x’. 

Usually, the preceding scaled problem is amenable to be solved, compared to 
the original. It will be seen that such doing enhances pivot column selection, 
consequently reducing the number of iterations required. 

An analogous scaling is to balance differences among rows. Denote row i of A 
by d} and introduce 


1/|Idi ll 


1/\|d 
re (Mea | a 


1/\ldml 
Then the standard LP problem can be converted into 
: _ ,T 
min f =c’x, (4.6) 
s.t. (D2A)x = Dob, x >0, 
where each row of the coefficient matrix D2A has norm 1. 

Professional codes for handling large-scale sparse problems generally use more 
sophisticated scaling methods. MINOS applies an iterative procedure based on 
Fourer (1979), which alternately balances nonzeros of rows and columns to obtain 
scaling factors by geometric mean. After the scaled problem is solved, relevant 


transformations are carried out to recover an optimal solution of the original. More 
precisely, the main steps are put in the following algorithm. 


Algorithm 4.2.1 (Scaling) Given a scaling tolerance scltol< 1. In the following, 
qj,; is the nonzero entry of A. 


. Compute aratio = max; {max; |q;,;|/ min; |a;, ||}. 

. Divide rows i = 1,...,m of Ax = b by ,/min; |a;, ;| x max; |qj, ||. 
a : A : 

. Divide columns i = 1,...,n of (4) by ,/min; |a;,;| x max; |a;, |. 

. Compute sratio by formula in step 1. 

. Return if sratio => scltol * aratio; else, go to step 1. 


nb WwW Ne 


As for more general LP problems, scaling factors for bounded variables, free or 
fixed variables should be determined differently, as omitted here. 

The simplex method is not invariant to different scaling methods. The effect of 
scaling is problem-dependent and is difficult to make a theoretical analysis, which 
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is not clear what kind of scaling is the best. At any event, scaling is indispensable 
in real applications. Even simple scaling would improve software’s performance 
significantly. Without scaling, only a few real-world problems can be handled. 


4.3. LU Factorization of Basis 


For large-scale sparse computation, it is important to maintain the sparsity of 
matrices and vectors in the solution process. To this end, this section presents a 
product form of the inverse of basis first and then turns to LU factorization of basis 
and its sparsity version, in particular. 

The simplex algorithm 3.3.1 and others utilize the inverse of basis explicitly and 
update it iteration by iteration. This is just for the convenience of expression, not 
suitable for sparse computation, since the inverse would be very dense even though 
the basis itself is sparse, as is shown by the following instance (Orchard-Hays 1971). 


Example 4.3.1 Itis verified that the inverse of matrix 


1S 


=f 7 4.211 
B=) 1 St 1 tt =1 }, 
= a oe | 
{ =4 1-214 


all of its entries being nonzero. 


That an original zero entry becomes nonzero is said to be fill-in. In the preceding 
instance, a large proportion of fill-ins lead to the inverse losing sparsity completely. 
An approach to bypassing this difficulty is to express inverse in product form of 
elementary matrices. 

As it is known, Gauss—Jordan elimination turns basis B to unite matrix. 
Such doing amounts to premultiplying B by a sequence of Gauss transformation 
(elementary) matrices E),--- , Ej such that 


En: - Ei B=T,. 
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Therefore, the product form of B a follows, 1.e., 
Bo! = Em---E}. 


Then, all operations involving B~! can be carried out through the factors. Instead of 
B~' itself, consequently, there is only a need for storing elementary matrix factors, 
which is usually sparse, relative to B~! itself. It is noted that only a single vector 
needs to be stored for each factor since the elementary matrix differs from a unit 
matrix by a column only. 

The preceding approach was considered in the early days of the simplex method. 
We will not go into detail but turn to a more desirable and commonly used 
alternative, i.e., the LU factorization. 

Concerning the computation in the simplex method, the inverse itself is not a 
goal. What one needs are vectors 


ag = Bae, and y= BoTcr, 
which can be obtained by solving the following two systems, respectively: 
Bag = dq, and = By =cg. (4.7) 


And a linear system can be solved through LU factorization of the coefficient matrix 
efficiently. In particular, there are sparse matrix techniques that solve very large 
sparse systems. 

Assume that the LU factorization B = LU is available, where L is a unit lower 
triangular matrix (with all diagonals 1), and U is an upper triangular matrix. Then, 
the first system of (4.7) is converted into the following two triangular systems: 


Lu=dg, Udg =u, (4.8) 
and the second to the two triangular systems below: 
Ulv=cg, L'y=v. (4.9) 


It is an easy task to solve such a system, as only m?/2 multiplications are required. 

Matrix factorization is closely related to elimination. The LU factors of a matrix 
can be obtained by Gauss elimination. In each step, Gauss elimination annihilates 
nonzeros below the diagonal in a column, which amounts to premultiplying the 
matrix by a Gauss transformation. 

Assume that at the end of the (k — 1)th step (k < my), there are Gauss 
transformations G1,--- , Gx—1 such that the entries below the diagonals of the first 
k — 1 columns are zero, 1.e., 


BE-D B&D 
BY 2G RS 1 oy), (4.10) 
Boo 
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where Boo is upper triangular of order k — 1. Assume that 


(k-1) (k—1) 
eee 


hg km 
BM. ae (4.11) 
He? a 
and that be # 0. Thereby, the following factors are well defined: 
lig =b4? 00? GRAAL ym. (4.12) 
It is clear that the Gauss transformation 
Ggpaianen. BS Ost Ones bai) (4.13) 


k 
satisfies 
(k) p(k) 
B® = G,Be-) = [ Bir Bia 
k (ky J> 
By4 


where Be is upper triangular of order k. Thus, the kth step of Gauss elimination is 
complete, provided the kth diagonal (pivot) is nonzero. In this manner, the sequence 


G,--+- ,Gm_, of Gauss transformations can be determined such that 


BD = Gy_1---GiB 5U 


is an upper triangular with nonzero diagonals. 
It is verified that G;! = 14h e.. Then, the following LU factorization follows: 


B=LU, 
where 
m-1 m—-1 
L=G;!---Gy), = [] +a ef) = 14 0 aed = 14h, ..., 2", 0) 
k=1 k=1 


is unit lower triangular. The preceding LU factorization has a practicable form, i.e., 


LOB=U, L'=Gyeye Gi. (4.14) 
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Therefore, (4.8) becomes 
Udg = Gm-1-++ Gidg, (4.15) 
and (4.9) does 
Ulv=cr, y=Gi-.-Gl_yv. (4.16) 


Consequently, there is no need for an explicit expression of L~!, but holding Gauss 
factors Gx, k = 1,...,m — 1, is enough. 

It is often the case when L~! is dense even though the Gauss factors are sparse. 
Expressing L~! in a product form is hence suitable for sparse computation. The 
required storage of Gx is not high, since only the nonzero components of h 
are stored. In dense computations, these nonzeros are usually stored in the lower 
triangular part of the matrix, while U is stored in the upper triangular part. 

It is noticeable that the LU factorization of a matrix is not always a clear 
road. The existence condition is that the principal submatrices are all nonsingular, 
guaranteeing a nonzero pivot in each step (Golub and Van Loan 1989); e.g., there is 
no LU factorization for the following simple nonsingular matrix: 


(11) 


because the first diagonal is 0, not eligible to be a pivot. 

Fortunately, there exists LU factorization for a permutated matrix, resulting from 
properly rearranging rows and columns, and such doing does not matter essentially 
but amounts to exchanging the order of equations/variables. 


4.4 Sparse LU Factorization of Basis 


On the other hand, rearranging rows and columns does affect the sparsity of Gauss 
transformations significantly. Let us look at a simple example. 


Example 4.4.1 Use Gauss elimination to convert the matrix 


into upper triangular. 
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Answer 
1 DAS 2. 
Gi =] -3/21 —_— G|B= —6-3 ], 
1/2 1 De 32. 
1 24 2 a 
G2 = 1 —_— G2G,;B= —6 2] =U. 
1/3 1 1 


The right-hand side U is already upper triangular. Vectors h,h2, related to the 
Gauss transformations, may be stored in the lower triangular part of U: 


2 42 
—3/2 -6 -2 
(J t/3-4 


It is seen that the resulting matrix is filled with nonzeros. Let us now rearrange rows 
and columns, according to the order of row indices 2,3, | and of column indices 
1,3, 2 as follows in advance: 


3 
B={|-11 
224 
Then carry normal eliminations in order: 
1 3 
Gi=] 1/3 1 —_ GiB= 1 ; 
—2/3 1 24 
1 3 x 
G2= 1 — G2G|B = 1 =U. 
—24 4 


Vectors hi, ho related to the Gauss transformations are stored in the lower triangular 
part of U, ie., 
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It is seen that the resulting matrix involves 3 zeros, just as the original B, and hence 
no fill-in is introduced. This is the case because the rearranged matrix is lower 
triangular, which is conducive to Gauss elimination. 


For the instance below, the left-hand side is the original matrix, where the symbol 
“x” denotes a nonzeros entry. If classical Gauss elimination is carried out, its 
sparsity is completely lost. But if rows and columns are rearranged as the matrix 
on the right-hand side, no fill-ins will occur at all. 


xxx xk xx x x 

x xX x x 

x x x x 
—_— 

x x x x 

x x x xX 

x x x XX X X X 


Note that the right-hand side matrix is close to lower triangular, with the bratty bump 
in the right-most column. 

The so-called preassigned pivot procedure (P*) and partitioned preassigned 
pivot procedure (P*) describe how to rearrange rows and columns for triangular- 
ization (Hellerman and Rarick 1971, 1972). Later, a stable revision, P5, of P4 
was proposed (Duff et al. 1986). All such types of approaches provide the LU 
factorization of some permuted version of a matrix, 1.e., 


L-'PBQ=U, L7'=Gyn-1-:-Gi, (4.17) 


where P and Q are the permutation matrices (resulting from the unit matrix by 
permutation). By (4.17 ), it is easy to convert the two systems in (4.7) into the 
following equivalent forms 


Uw =Gm-1°--GiPdg, ag = Qw (4.18) 
and 
|v. (4.19) 


A combination of such approaches and the classical Gauss elimination perform very 
well in practice. We will not go into detail; though, interested readers are referred to 
the related literature. 

We prefer another approach to select pivots in the elimination process, rather 
than via preassigning order. In this aspect, Markowitz’s rule (1957) is especially 
successful in practice. 

Assume that at the end of step k — 1, Gauss elimination generates (4.10)~(4.11). 
Currently, only the (m —k-+ 1) x (m —k + 1) southeast corner submatrix is active, 
and hence, the elimination proceeds only within it without touching the other part. 
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At step k, the standard Gauss elimination would take i as pivot, provided it is 
nonzero. 
Denote by ad the number of nonzeros of row i of Bo, and by cf the number 


of nonzeros of column j. Markowitz selects a pivot, within nonzeros of ees such 


that fag —1)x (ci — 1) is the minimum. The pivot chosen is then moved to the kth 
diagonal’s position by row and column permutations, and step k is complete. 

It is noted that (rf —1)x (ck — 1) rather than rk x e is used; therefore, a row 
or column involving only a single nonzero is selected, if any, and hence, no fill-in 
occurs at step k. 

However, the so-called zero or nonzero only has conceptual meaning, since a 
pivot too close to zero would lead to computational failure. Considering numerical 
stability, the pivot chosen should not be too small, compared with other nonzeros. 
Therefore, there is a contradiction between sparsity and stability. In the following 
rule, the two aspects are balanced to some extent. 


Rule 4.4.1 (Markowitz Pivot Rule) Given a real number a such that0 < o < 1. 
Select pivot ba such that 


: k- eS, 
(rp — VD) x (cg — 1D) = min(rf — 1) x (cf - 1) | 1b? | S08, if = ky), 
(4.20) 
where 
6 =max{|b~ | |i, j =k,...,m}. (4.21) 


ij 


It is clear that a large o value contributes to stability, while a small one does to 
sparsity. Value o = 0.5 or so seems to be acceptable. To avoid going through every 
nonzero of the active submatrix, on the other hand, some variants of the preceding 
rule are designed, in which only a part of rows/columns is involved. 

Such kind of approaches lead to the following LU factorization: 


Gm-1Pm-1-++GiPiBQi--: Qmn-1 = U, (4.22) 
where P;, Q;,i = 1,...,m — 1, are permutations. Setting 

P= Pm-1-+: Pi, Q=Q1--:Qm-1, 
then from (4.22), it is known that the pivot determinations and row/column 


permutations, carried out alternately, have the same effects as the corresponding 
prepermutations followed by the standard factorization, i.e., 


L'PBQ=U, L7!=Gm—-1Pm—1-:-GiP\P'. 
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Therefore, the two systems in (4.7) can still be, respectively, converted into 
Uw = Gm-1Pm-1°::GiPidg, dg = Qw 
and 
UTv=Qlcp, F= Pi--GT-+- Pi Gy_yv. 


It is favorable that there is no need for moving around related entries for 
permutations in the process; two integer arrays are enough to record the positions of 
pivot rows and columns. 


4.5 Updating LU Factors 


The LU factorization of a basis from scratch usually requires a large number 
of operations. Fortunately, it is unnecessary to do so in each iteration. Once 
an initial basis is factorized, subsequent LU factors can be updated iteration by 
iteration. A basis is refactorized only when the number of iterations exceeds some 
predetermined bound, and hence, the LU factors are unacceptably inaccurate or 
dense, or the sequence of the factors is too long to store (Sect. 4.1). 

Only one column of basis is different from that of its predecessor; therefore, it is 
possible to obtain LU factors from the old ones cheaply. In this section, we present 
updating approaches commonly used in practice. 

Assume that the LU factorization of the current basis is 


L'B=U, L7!=G,P,---GiP} (4.23) 

and that a pivot column index q and row index p have been determined. 
The first updating approach, proposed by Bartels and Golub (1969), drops the 
pth column aj, from B, then moves the p + | through mth columns forward by 


one column distance, and adds a, at the end as the mth column, resulting in the new 
basis 


B=(a@j,..., jy 1s 4jnyireees jn Aq) 
Consider the matrix 
A;-1F8 = -1 =] = 24 
H=L B= (L Bjyrs* »L jy. Lb Binge" JL A jn» L aq). 
Note that the last column aq = | aac of the new basis is available since it was 


computed as an intermediate result when solving Bag = ay. It is also seen that 
HT may result from dropping the pth column from the upper triangular matrix U, 
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moving forward the p+ 1 through mth columns by one column position, and putting 
dq as the last column. Therefore, H is an upper Hessenberg matrix with nonzero 
subdiagonals in the p through m — Ith columns, the pattern of which looks like: 


x xX xX xX 
x xX xX 
x xX 

x 


x * X X X X 

x xX * X X X X 

x xX X * X X X &X 

x xX X X * X X X X&X 
x xX X X X * X X X xX 
x xX KX X KX * KX XK XK X& 


Then eliminate H’s nonzero subdiagonals in the p through m — Ith columns, in 
turn, by elementary transformations to lead to upper triangular. 

However, the preceding process is impossible when some diagonals are zero. 
Bartels—Golub’s approach overcomes this difficulty by exchanging rows: in the 
elimination process it takes a diagonal as pivot whenever its absolute value is large 
enough, compared with the corresponding subdiagonal; otherwise, it interchanges 
the two rows where the diagonal and subdiagonal are located. Such doing leads 
to better numerical stability, though exchanging rows frequently could degrade 
efficiency. 

As a variant of Bartels—Golub approach, Forrest and Tomlin (1972) moves the 
pth row of H (marked by x) to the bottom (the p through m — 1th rows is moved 
up by one row position). The resulting matrix differs from upper triangular only in 
p through (m — 1)-indexed possible nonzeros in row m: 


x xX xX xX 
x xX xX 
x xX 

x 


x xX X XK X 

x xX X XK X X 

x xX KX KX X XK X 

x xX X X KX X XK X 
* XX X KX KX KX KX K XK X 
* X X KX KX KX KX KX XK X 


* OK OK Ok 


Doing so amounts to premultiplying H by a permutation, say P. Assume that the 
resulting matrix is H = PH, in which the first nonzero in row m is p-indexed, 1.e., 
hmp # 0. Then, taking diagonals in the p through (m — 1)th columns as pivots, 
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we eliminate nonzeros in row m, in turn, by elementary transformations. This gives 
an upper triangular matrix U. As all diagonals are nonzero in this case, no row 
exchange is needed theoretically. For sake of numerical stability, however, if the 
ratio of the nonzero in the mth row to the associated diagonal is greater than o 
in magnitude, interchange the mth row and the row where the diagonal is located 
before eliminating. For this purpose, o = 10 or so seems to be acceptable. 

Assume that there is no need for such a row exchange. Take elimination of h mp 
as an example. This doing amounts to premultiplying H by elementary matrix 


Pp? 


Gp=I—hPel, h? =(0,--> ,0,hmp/h?)'. (4.24) 
———— 


m—1 


Consequently, the LU factorization of the new basis B follows, i.e., 


EL 1B=0, £1 =Gn1---G,pPL. (4.25) 


In view of that the lower end component of vector dg is often zero in sparse 
computations, Reid (1982) suggests not putting a, as the last column of the new 
basis, but inserting it to as the (t— 1)th column instead, if its first nonzero component 
(counting from the end) is in row f, that is, a; ,g: 


Pp t 
x xX xX xX XK KKK XK XK 
x xX xX xX KKK XK XK 
x xX xX KX KK XK XK 
x xX xX XK XK XK XK 
H= * OK OK OK OK O& P 
x xX xX XK XK xX 
x XX XK xX 
x X XK xX ' 
x xX 
x 
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After that, remove the pth row, then move the p + | through fth rows up a row 
distance, and put the original pth row as the new th row, ie., 


Pp t 
x XX XX X X X X X 
x X X X X X X X X 
x xX XX X X X X 
x XX X X X X 
H= xxxxxx |? 
x xX X X X 
x xX X X 
* OK OK OK OK OX : 
x xX 
x 


Finally, eliminating nonzeros in the p through t — 1th columns of the tth row leads 
to the wanted upper triangular matrix. 

At present, Forrest-Tomlin’s approach and its variants, such as Reid tactics, are 
commonly used for updating LU factors in practice. Suhl and Suhl (1993) give very 
detailed considerations on the implementation. 


4.6 Crash Procedure for Initial Basis 


The simplex method and its variants require an initial basis to get them started. 
There is no doubt, quality of the initial basis greatly affects the required number of 
iterations and storage. 

There are several so-called crash procedures for determination of initial basis, for 
which being sparser, as well as tentatively reducing subsequent iterations required. 

Let us bring up the crash procedure of MINOS, designed for solving general 
linear and nonlinear programming problems. Once the original data is entered, it 
introduces a so-called logical variable to each equality of the constraint system to 
convert the problem into a bounded-variable one with a zero right-hand side. As a 
result, the coefficient matrix of logical variables forms a unit matrix, which is taken 
by a crash option of MINOS as an initial basis. 

Considering that an initial basis will be LU-factorized, some crash options of 
MINOS select basic columns to form a nearly lower triangular matrix to limit the 
number of fill-ins. A standard option of crash includes 5 stages. At the first stage, all 
free logical variables enter the basis, the number of which is usually less than m. At 
the second stage, other free variables (or those with very wide bounds) enter. At the 
third stage, columns having only a single nonzero in the active submatrix enter. At 
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the fourth stage, columns having two nonzeros do. If the number of selected basic 
columns is still less than m, some logical variables are taken to form the end basis, 
at the fourth stage. 

It is “ideal” that the generated initial basis, say B, happens to be optimal, and 
what to do is to solve system Bxg = Db only. This is of course impossible in 
general. But it would be possible to be close to optimal to some extent. In this 
respect, it seems to be favorable to determine basis based on the most-obtuse- 
angle heuristics: according to Proposition 2.7.1, lower pivoting-indexed inequalities 
tend to be satisfied as equalities at an optimal solution, that is, the associated slack 
variables should be taken as nonbasic variables or larger pivoting-indexed variables 
taken as basic ones. Let us illustrate by the instance below. 


Example 4.6.1 Determine an initial basis by the most-obtuse-angle heuristics: 


min f = 3x; — 2x2 + x3, 

s.t. xy + 2x2 -—x3-— 2x4 = 1, 
xy tx2-—x34+ x5 =3, 
—x, +4x2+2x3+%x6 = 2, 
xj = 90, jJ=1,---,6. 


Answer The problem can be converted into 


min f = 3x; — 2x2 + x3, 

s.t. x4 = —l4+ 4x, +2x2-—2x3> 0, 
X5=3—-—x1—x2+ x3 = 0, 
x6 = 24x, —x2- 23 >0, 
X1,X2,x3 > 0. 


The following table lists pivoting-indices a; for variables: 


Variable | Constraints ‘1 aj 

Pal x, >0 —3.00 
X6 x1 —%2—x3 > —2 —2.31 
x3 x3 >0 —1.00 
x5 —x, —x2 +43 > -3 0.00 
x4 Xj + 2x2 -—x3 > 1 1.63 
x2 x2 >0 2.00 


The first 3 pivoting-indices are the smallest, corresponding to constraints 
whose gradients form the most-obtuse angles with the negative objective gradient. 
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Therefore, correspondingly x1, x3, x6 are taken as nonbasic variables. This leads to 


1 


2—-10\— 00 1 
B={2, 4, 5}, N={1,3,6}. B'=[{1 o1] ={[-10 2 
1 00 orS) 


Therefore, if one solves Bxg = b, the basic solution follows, that is, 


00 1\ /1 y) 
ig =B'b=|[-10 2])/3]=[13]2=0, y= (,0,0)', 
O1<1) \2 1 


the optimality of which can be verified by calculating the associated dual solution, 
1.€., 


60. 1" fo 0 
j3=Blcp=|[-10 2 OP 1S) 
O1=1 0 0 
3 1-10\' /o j 
Iv=cyn-N'y=]1]-] 1-10 2)=[3] 20. 
0 aq. 4 0 0 


Therefore, there is no need for any iteration. 

It is interesting that for small textbook instances, an optimal basis can often be 
obtained in this way; even if it is not the case, only a few iterations are required 
subsequently to achieve optimality. It is noted however that the approach has little 
effect if differences between pivoting-indices are insignificant. For large-scale LP 
problems, there is a serious conflict between basis’ sparsity and the most-obtuse- 
angle heuristics that are not easy to balance. Interested readers are referred to the 
related literature, e.g., (Li 2004; Hu and Pan 2006, 2008). 


4.7 Harris Rule and Tolerance Expending 


For the success of the simplex algorithm, controlling the size of the pivot is not only 
important but also cumbersome. A code based on the standard row rule (see step 6 
of Algorithm 3.3.1) is not practical. It can solve some very small textbook problems 
because such a pivot could be too close to zero to ensure numerical stability. 
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It is necessary to impose restriction on pivot size. At least, the parameter tolpiv 
should be such that 


€é, ¥ 107! <tolpiv < 1077 (4.26) 


(see Sect. 4.1). It is still unsatisfactory though. In fact, tolpiv = 10~!! is too small 
for many problems, while tolpiv = 107’ is often too large. It is difficult to find a 
tolpiv value that is suitable for all LP problems. In this aspect, Harris (1973) rule 
with tolpiv satisfying (4.26) seems to be feasible overall. 

She designed a so-called two-pass procedure, which passes m rows twice in each 
iteration. In the first pass, a prospective stepsize a; is determined by allowing the 
bounds violated by a small tolerance 5 (0 < 56 < 1 ), which is equal to the current 
feasibility tolerance (this will be clear a little later). In the second pass, a pivot row 
is selected from rows, corresponding to real stepsizes no more than a. The rule can 
be formulated as follows. 


Rule 4.7.1 (Harris Two-Pass Row Rule) First, determine 
ay = min{(b; + 6)/diq | dig > tolpiv, i=1,..., m}, (4.27) 
and then select pivot row index 
p € arg max{Gj q | bi [dig <1, Gig > tolpiv; i= 1,..., m}. (4.28) 


The preceding rule widens the scope of pivot row candidates so that a pivot with 
a larger size is chosen to improve numerical stability. In addition, geometrically the 
gradient of the nonnegative constraint x;, > 0 and the descent edge indexed by q 
form a larger angle, which is conformable with the heuristic characteristic of the 
optimal solution. Harris’ two-pass rule performs well in practice and is at present an 
indispensable part of simplex LP codes. However, the stability difficulty still occurs 
occasionally. 

On the other hand, the conceptual stepsize a = b p/@pq seems to be another 
source of troubles. Even if pivot dpq has a normal size, there is still a somewhat 
sticky problem when b, is slightly less than zero, leading to a little increase of 
objective value. Essentially, this is an effect of degeneracy or near degeneracy. To 
guarantee a positive stepsize, and hence a strict decrease in objective value, Gill 
et al. (1989) suggest a tactic termed tolerance expanding. 

In each iteration, the current feasible tolerance 6 is increased by a tiny amount, 
say T, 1.€., 


6=6+T. 
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Assuming that upper bound featol of 6 is set to 10~° (see 4.1), and the initial value 
is set to 


69 = 0.5 x 107°, 
then the increment 


(0.99 — 0.5) x 1076 


= 0.49 x 107!° 
10000 tee 


seems to be acceptable. A positive stepsize is then set by 
a = max{bp/apq, t/Gpq}. (4.29) 


After every 10000 of iterations, 6 = do is reset, the basis matrix is refactorized, 
and the associated solution is recalculated. Of course, a phase-1 procedure will be 
carried out again if the solution becomes infeasible. After optimality is achieved, 
in addition, the solution process has to be restarted by setting 6 = do to ensure the 
quality of the final optimal solution. 


4.8 Pricing for Reduced Cost 


All pivot rules are closely related to the reduced costs. Availability of all or a part of 
the reduced costs is a prerequisite for the determination of the pivot. This section is 
devoted to approaches to calculating them or pricing. 


We might as well assume that the current basis is B = {1,..., m}, and nonbasis 
is N = {m+ 1,...,n}. The classical pricing method is based on the following 
formulas: 

Bly =cz, ZN =cy —N'y. (4.30) 


We will consider several alternative schemes, some of which are being used in 
practice widely. 

Assume that a column index g € N and row index p € B have been determined. 
Let the prime mark the quantity associated with new basis, compared with those 
associated with old basis, e.g., 


B’ =(B\p)Uq, N’=(N\q)Up. (4.31) 
Then, the following relation holds: 


B’ = B+ (aq —ap)es, (4.32) 
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and y’ satisfies 

By! =cp. (4.33) 
If h denotes the solution to the following system: 

B"h = ep, (4.34) 
then h’ satisfies 

Bh! = ey. (4.35) 


Thereby, the following recurrence formulas can be shown: 


y =ytzgh’, (4.36) 
h = Gp gh’, (4.37) 
Ziyr = ZN — Zg NTN’, (4.38) 
= Zur — (2q/Ap.q)N A, (4.39) 


where p,q is the pivot, i.e., the pth component of the solution to Bag = dg, which 
can be calculated independently of pricing. Note that z,» = 0 in the preceding 
formulas. 

As an alternative scheme, the reduced costs can be calculated based on (4.38) 
or (4.39). Zoutendijk (1960) uses formulas (4.38) and (4.35). Bixby (1994) 
uses (4.39) and (4.34). An advantage of the latter is that solution h/h’ to (4.34)/(4.35) 
is usually much sparser than y and especially suitable with the steepest-edge rule 
(Part ID), which also needs to solve (4.34)/(4.35). 

Nevertheless, recurrence is not suitable for partial pricing. In contrast to the 
preceding schemes, instead, Tomlin uses (4.36) and (4.35) to compute the simplex 
multipliers in recurrence manner, which is amenable to partial pricing. 

As in the classical pricing, the preceding schemes need to solve two triangular 
systems. Assume that LU factorization with row and column exchanges of B is 


PBQ=LU, (4.40) 
where L is the unit lower triangular, U is the nonsingular upper triangular, and P 
and Q are permutations. Recall that each system involved in simplex iterations can 
be converted into two triangular systems. For Bag = dq, €.g., one may solve 


Lu= Pag, Uu=v, (4.41) 


and set Gg = Qu. It is similar to solve (4.34)/(4.35). 
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Hu and Pan (2008) modified Tomlin’s scheme so that only a single triangular 
system needs to be solved for pricing. It is found later that Goldfarb (1977) proposed 
the same approach along another line earlier without offering any numerical results. 

In the following, we will focus on this scheme and report encouraging results, 
obtained in extensive computational experiments. To this end, first, we derive the 
Bartels—Golub update in a way, slightly different from that in Sect.4.5 (Forrest— 
Tomlin’s variant). 

From (4.32) and (4.40), it follows that 


PB'Q = LU + P(aq — ap)e,O, 
= LU + (L7' Pag — L~' Pap)e},Q). 
By (4.40) and Be, = Gp, it holds that 
Pa, = PBep = LUQ' ep. 
Besides, it is clear that there is an integer | <r < m such that 
er = Q' ey. (4.42) 

Combining the preceding 3 expressions leads to 

PB'Q = LU + (L™' Pag — UQ"ep)e,Q) = LR, (4.43) 
where v satisfies the first triangular system of (4.41), and the matrix 

=(U+(v—Ue,)er 


is upper triangular, except for the rth column, which results from U by replacing the 
rth column by v. Move the rth column backward to the end position and the (r + 1) 
through mth columns forward one column. Then put the rth row to the bottom and 
the (r + 1) to mth rows up one row. If O denotes the associated permutation, then 
the resulting matrix oT ROi is upper triangular, except for entries in the r through 
m — 1th columns in the mth row perhaps being nonzero (m > 2). These entries 
can be eliminated by a series of Gauss transformations; while proceeding, exchange 
the mth row and the row at which the diagonal locates whenever a diagonal is too 
small in magnitude, compared with the corresponding nonzero entry. Thus, there 
are permutations P; and lower triangular matrices L;,i = 1,...,sd <s<m-—r) 
such that 


U’ (4.44) 
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is nonsingular upper triangular. On the other hand, it is clear that 
L'’=LOP}L,---P!L, (4.45) 
is lower triangular with row exchanges. From (4.44), (4.45), and (4.43), it is verified 


that the new basis B’ has the following LU factorization with row and column 
exchanges: 


PBO=L'U', (4.46) 
where 
P=P, Q=00. (4.47) 


This process simplifies computation of simplex multipliers. 


Theorem 4.8.1 Let y be a solution to B'y = cg and 2q be reduced cost related to 
column index q. If w' is the solution to 


L™ Pw! = (1/u) mem: (4.48) 


where U' jm is the mth diagonal of U', then y' = y + zqw’ is the solution to (4.33). 


Proof Based on relations between quantities, related to the old and new bases, it is 
only required to show that w’ is the solution to (4.35). 
Premultiplying (4.48) by U™ gives 
COE Pi = fa eins 
which, combined with (4.46), (4.47), and Ue, = u' mem; leads to 
or OTB w’ = em. 
Then by (4.42) and the definition of permutation O, it is known that 
e, 00 = eo = er. 
Finally, it follows from the preceding two expressions that 
fi — Q0em = 000" Qe), =€Cp.- 
| 


This theorem says that after new LU factors are obtained by Forrest—Tomlin’s 
update, one can solve the triangular system (4.48) for w’, calculate the new simplex 
multipliers y’, and then calculate the new reduced costs by (4.38). 
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Table 4.1 Iteration and time ratios of MINOS 5.51 to NEW 


Small Medium Large Ken. BPMPD Total average 
Problem (38) (41) (17) (16) (17) (129) 
Iterations 1.00 1.01 1.12 1.05 1.12 1.10 
Time 1.03 1.17 1.28 1.28 1.23 1.24 


Our numerical experiments involve the following two codes (Hu and Pan 
2008): 


1. MINOS 5.51. 
2. NEW: MINOS 5.51 with the presented approach to pricing. 


The preceding codes both adopted default sectional pricing options (partial price 
10). 

There were 96 Netlib, 16 Kennington, and 17 BPMPD test problems involved 
(Appendix C: Tables C.1—C.3). All the Netlib problems fall into three sets: Small 
(38), Medium (41), and Large (17) (in the order of increasing m +n). Table 4.1 lists 
the total iteration and time ratios of MINOS 5.51 to NEW. 

From the preceding table, it is seen that the presented pricing scheme outper- 
formed the standard scheme with each set of problems (except for small Netlib 
problems), in terms of either iteration or running time. The overall iteration and 
average time ratios attain 1.10 and 1.24, respectively. 
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Chapter 5 ® 
Duality Principle and Dual Simplex crests 
Method 


Duality is an essential part of LP theory. It is related to the special relationship 
between one LP problem and another, both of which involve the same data (A, b, c), 
but positioned differently (except for the self-duality, see below). The former is 
referred to as primal problem, and the latter as dual problem. Based on duality, once 
a primal problem is solved, so is the dual problem, and vice versa. This chapter is 
devoted to such topics as the relationship between their feasible regions as well as 
optimal solutions, optimality condition, dual simplex algorithm in tableau form and 
in revised form, shadow price, and dual elimination. Before making notes on the 
duality in the final section, we introduce a special bilevel LP problem together with 
a related approach to intercepting the optimal set. 


5.1 Dual LP Problem 


Now call the standard LP problem (1.8), i.e., 


(P) min f =c!x 


5.1 

s.t. Ax = b, x>0 ey) 
primal problem, and the following problem 
_— pT 

(D) max g=b'y (5.2) 


S.t. Aly+z=c, z>0, 


dual problem. There is 1-1 correspondence between their variables/constraints. 
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(y, z) satisfying AT y + z = c is called dual feasible solution. The set 
D={VV,. DER" xR" | Alyt+z=c, z= 0} 


is called dual feasible region, which includes all dual feasible solutions. 
Given basis B, setting zg = 0 in Bly + ZB =p gives 


y=Bfcg, Z=0, %y=cn—N'y, (5.3) 


called dual basic solution. z is just the reduced costs; and y the simplex multiplier. If 
Zn = O, (¥, Z) is a dual feasible basic solution, corresponding to a vertex in D. For 
simplicity, thereafter zy alone is often said to be dual basic solution. In particular, 
(y = 0, z =c) is a dual feasible solution if c > 0. 

The following equivalent form of dual problem (5.2) 


(D) max g=bry, 


5.4 
s.t. Aly <e¢ on 


is useful. Problems (5.2) and (5.4) will be regarded as the same. 
As it can be converted into a standard one, any LP problem corresponds to a dual 
problem. By introducing slack variables u > 0, e.g., the problem 


max C Ty 


st. Ax<b, x>0 Ga) 


can be turned to the standard problem 


min —c!x 


st. Axtu=b, x,u>0 
and the dual problem of which is 
max bly’ 
a ATS. 3 si = 
s.t. ri? =e) 
Then, setting y = —y’ turns the preceding to the dual problem of (5.5), i.e., 


min bly 
st. Aly>c, y>0. 
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The correspondence between the primal and dual problems is summarized into 
the table below: 


Primal problem Dual problem 
Objective function min Objective function max 
Variables | Nonnegative | Constraints < 
Nonpositive > 
Free = 
Constraints “> | Variables =|. Nonnegative 
< Nonpositive 
= Free 


Note In applications of the preceding table, sign restriction is not handled as a 
constraint but attributed to the associated variable. 
For example, the so-called bounded-variable LP problem 


(5.6) 


can be transformed into 


x+s=u, (5.7) 


(5.8) 


The so-called self-duality refers to a special case when the dual problem of an 
LP problem is just itself. Combining primal problem (5.1) and dual problem (5.4), 
we construct the following problem: 


min c!x — bly 
s.t. Ax 


ll 
= 
be 
IV 
= 


(5.9) 
Aly 


IA 
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According to the preceding table, the dual problem of it is 


max bly + cly 
s.t. Aly SC; 
Av =—b, v <0. 


By setting v = —x and handling the objective function properly, the preceding 
can be transformed into the original problem (5.9). Therefore, (5.9) is a self-dual 
problem. 


5.2 Duality Theorem 


This section only focuses on the nonsymmetric duality of (P) and (D), as obtained 
results are valid for the general case. 


Theorem 5.2.1 (Symmetry) The dual problem of the dual problem is the primal 
problem. 


Proof Introduce the slack variable vector u > 0 to the dual problem (D), and make 
the variable transformation y = y; — y2 to convert it into 


max b'(y; — y2) 
s.t. AT(y1 — y2) tu =c, yi, y23u > 0 


or equivalently, 


min (—b7, bT, O)(yT, yd, ut)? 
st. (AT] —ATI DO, yu =c, yey,u>0. 


The dual problem of the preceding is 


max c! x’ 
A —b 
s.t —A |x’ < b], 
I 0 
that is, 
max c! x’ 
s.t. Ax’ = —b, x’ <0, 


which becomes (P) by setting x’ = —x. Oo 
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The preceding says that any of the primal and dual problems is the dual problem 
of the other. So, the two are symmetric in position. This is why any fact, holding for 
one of them, has its counterpart for the other. It is important that there is a special 
relationship between feasible/optimal solutions to the two problems. 


Theorem 5.2.2 (Weak Duality) [f x and y are feasible solutions to primal and 
dual problems, respectively, then clx > bly. 


Proof Premultiplying c > A'y by x > 0 gives c'x > y'Ax, substituting b = Ax 


to which leads to clx > bly. oO 


According to the preceding, if there are feasible solutions to both primal and dual 
problems, any feasible value of the former is an upper bound of all feasible values 
of the latter; on the other hand, any feasible value of the latter is a lower bound of 
all feasible values of the former. 


Corollary 5.2.3 [f any of the primal and dual problems are unbounded, there exists 
no feasible solution to the other. 


Proof By contradiction. Assume that there is a feasible solution to the dual problem. 
Then it follows from Theorem 5.2.2 that feasible values of the primal problem are 
bounded below. Analogously, if the primal problem is feasible, the dual problem is 
bounded above. Therefore, the claim is valid. oO 


Corollary 5.2.4 Let x and y be primal and dual feasible solutions, respectively. If 


c'x =b"¥ holds, they are optimal solutions. 


Proof According to Theorem 5.2.2, for any primal feasible solution x, it holds that 
c'x > b'y = c'x; therefore, x is a primal optimal solution. Similarly, j is a dual 
optimal solution. Oo 


Theorem 5.2.5 [Strong duality] If there exists an optimal solution to any of the 
primal and dual problems, then there exists an optimal one to the other, and the 
associated optimal values are equal. 


Proof Assume that there is an optimal solution to the primal problem. According 
to Theorem 2.5.5, there is a basic optimal solution. Let B and N be optimal basis 
and nonbasis, respectively. Then 

ch—chB 'N>0, B 'b>0. 
Thus, setting 


y= Bl cpg (5.10) 


leads to 
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Therefore y is a feasible solution to the dual problem. By (5.10), on the other hand, 
the feasible basic solution (¥g = B~'b, xy = 0) satisfies 


By Corollary 5.2.4, therefore, x and y are the respective optimal solutions to the 
primal and dual problems with the same optimal value. Moreover, it is known by 
Theorem 5.2.1 that if there is an optimal solution to the dual problem, so is to the 
primal problem, with the same optimal value. Oo 


Based on the strong duality, thereafter the primal and dual optimal values will 
not be distinguished. 

It is clear that if there is an optimal solution to one of the pairs of (5.1) and (5.4), 
so is the self-dual problem (5.9). Moreover, the optimal value of the latter is 
equal to zero, and the optimal solution of the latter gives the primal and dual 
optimal solutions to the pair. A variation of it will be used to derive the so-called 
homogeneous and self-dual interior-point method (10.4.4). 

In case when any of the primal and dual problems are infeasible, it can be 
asserted that the other problem is infeasible or unbounded. The computation would 
be finished then. In some applications, however, it would be needed to distinguish 
between the two cases. This can be resolved via the duality as follows. 

Assume now that the primal problem (5.1) is infeasible. To determine whether 
the dual problem (5.2) is infeasible or unbounded, consider 

min c!x 

s.t. Ax = 0, x > 0, ol) 
which has a feasible solution x = 0. Solve the preceding problem by the simplex 
algorithm. If (5.11) is unbounded, then the problem 


min 0 
st. Aly +z=c, z=0 


is infeasible (Corollary 5.2.3). Therefore, (5.2) is infeasible either. If an optimal 
solution to (5.11) is reached, then there exists an optimal solution to the preceding 
problem (Theorem 5.2.5), which indicates that (5.2) is feasible; therefore, it can be 
further asserted that (5.2) is unbounded. 

Using the duality, now we are able to prove Farkas Lemma 2.0.3 concisely. 


Lemma 5.2.6 (Farkas) Assume A € R'””*" and b € R". The feasible region P is 
nonempty if and only if 


b'y > 0, Vye{yeR™ | Aly = 0}. 
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Proof Consider the following LP problem 


min 0 


5.12 
s.t. Ax =b, x>=0, ween 
the dual problem of which is 
max bly’ 
5.13 
s.t. Aly’ <0. om) 


Note that y’ = 0 is a feasible solution to it, with the feasible value 0. 

Necessity. Assume that the feasible region P of (5.12) is nonempty; hence, 
all feasible solutions correspond to the same objective value 0. According to 
Theorem 5.2.2, for any y’ € {y’ € R”™ | ATy’ < O}, it holds that bTy’ < 0. 
By setting y = —y’, it is known that b’ y > O holds for y € {y € R”™ | Aly > O}. 

Sufficiency. Assume that for any y € {y € R” | Aly > O} it holds that 
b'y > 0. Then, for y’ € {y’ e R”™ | ATy’ < 0}, we have b'y’ < 0; hence, there 
is an optimal solution to (5.13). According to Theorem 5.2.5, therefore, there is an 
optimal solution to (5.12), which implies P being nonempty. Oo 


5.3 Optimality Condition 


From duality theorems presented in the previous section, it is possible to derive 
a set of conditions for primal and dual solutions to be optimal, which serve as a 
theoretical basis for LP. We consider the standard LP problem first and more general 
LP problems then. 

Assume that x and (y, z) are the primal and dual (not necessarily feasible) 
solutions, respectively. 


Definition 5.3.1 The difference c'x — by between the primal and dual objective 
values is the duality gap between x and (y, Z). 


Definition 5.3.2 If x™z = 0, x and (y, Zz) are complementary; if x + z > 0, in 
addition, the two are strictly complementary. 


The quantity x"z is termed complementarity residual. 


Lemma 5.3.1 The duality gap and complementarity residual of x and (y, z) are 
equal; x and (y, z) are complementary if and only if their duality gap equals zero. 


Proof Since x and (y, z) satisfy the equality constraints, it is verified that 


clx — bly =x'¢c= (Ax)! y = xl(c _ A'y) =x'yz, 
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A “>” or “<” type of inequality is said to be tightly satisfied if the “=” holds; it 
is said to be slackly satisfied if the “>” or “<” holds. 
In case when the nonnegative constraints are satisfied, the complementarity of x 


and (y, Zz) is equivalent to the satisfaction of 
xjzj =0, Vij haan (5.14) 


It is clear that for j = 1,...,n, it holds that x; = O/z; = Oif zj; > O/x; > 0). Ifa 
component of x/z slackly satisfies the associated nonnegativity constraint, therefore, 
the corresponding component of z/x must satisfy the associated nonnegativity 
constraint tightly. 


Theorem 5.3.2 (Optimality Condition for the Standard LP Problem) x is an 
optimal solution of the standard LP problem if and only if there exist y, z such that: 


(i) Ax=b, x > 0, (primal feasible) 
(ii) Aly +z=c, z= 0, (dual feasible) (5.15) 
(iii) x'z =0. (complementary) 


Proof Note that for x and (y, z), zero duality gap is equivalent to complementarity 
(Lemma 5.3.1). 

Sufficiency. By Corollary 5.2.4 and the equivalence of zero duality gap and 
complementarity, it follows from (5.15) that x and (y, z) are primal and dual 
optimal solutions, respectively. 

Necessity. If x is a primal optimal solution, it satisfies condition (i). By 
Theorem 5.2.5, in addition, there is a dual optimal solution (y, z) such that the 
duality gap is zero; hence, conditions (ii) and (iii) are satisfied. oO 


The following result is stated without proof (Goldman and Tucker 1956b). 


Theorem 5.3.3 (Strict Complementarity) [f there exists a pair of primal and dual 
optimal solutions, then there exists a strictly complementary pair of such solutions. 


The significance of the optimal conditions speaks for itself. As for method 
research, any type of optimality criteria in various contexts must coincide with these 
conditions. From them, it is understandable that LP methods always solve the pair 
of problems at the same time. For instance, once the simplex method reaches the 
primal optimal solution 


iso, ay = 0, 
it also gives a dual optimal solution 


y= B' cg, Zp = 0, Zy =cn — N'Y. 
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In view of the symmetry between primal and dual problems, moreover, it is not 
surprising why LP methods often present in pair: if a method solves the primal 
problem, there is another method that solves the dual problem, and vice versa. For 
example, the dual method matches the simplex method. 

The interior-point methods presented later often judge the degree of approaching 
optimality through the complementarity residual: the smaller the residual is, the 
closer to optimality; when it vanishes, the primal and dual optimal solutions are 
attained respectively. It is noticeable moreover that direct dealing with the optimal 
conditions as a system of equalities and inequalities can lead to some interior-point 
algorithms (10.4). Such algorithms usually generate a strictly complementary pair 
of optimal solutions in the limit, which is important for asymptotic analysis (Giiler 
and Ye 1993). 

For the bounded-variable LP problem (5.6), we have the following result. 


Theorem 5.3.4 (Optimal Condition for the Bounded-Variable Problem) x is 
an optimal solution of the bounded-variable LP problem if and only if there exist 
y, v, w such that: 


(i) Ax=b, l<x<u, (primal feasible) 
(ii) Aly-v+w=c,v,w,>0, (dual feasible) (5.16) 
(iii) (x —D'w =0, (u — x)'v = 0. (complementary) 


Proof It follows from (5.6), (5.7), (5.8), and Theorem 5.3.2. oO 
Consider the following more general minimization problem: 


min c™x 


17 
st. x EQ, an 


where Q C FR” is a convex set. 
Lemma 5.3.5 x* € Q is an optimal solution to (5.17) if and only if it satisfies 
chee") = 0, ¥ xen. (5.18) 


Proof Assume that x* is an optimal solution. If (5.18) does not hold, i-e., there is a 
point x € © such that 


cl (x —x*) <0, (5.19) 
then it holds that 


cx <clx*, (5.20) 
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which contradicts that x* is optimal. Conversely, assume that (5.18) holds. If x* is 
not optimal to (5.17), then there exists x € Q satisfying (5.20), which implies (5.19), 
contradicting that (5.18) holds. oO 


The preceding lemma says that a sufficient and necessary condition for x* to 
be an optimal solution is that there exists no feasible descent direction at x* with 
respect to the objective function; or geometrically, the angle between any feasible 
direction and the gradient of the objective is not obtuse. Note that, the lemma is 
actually valid for an arbitrary set Q, including the feasible region of the LP problem. 


Definition 5.3.3 Let x* be a feasible solution to an LP problem. Another LP 
problem generated by dropping all or part of inactive constraints at x* is said to 
be a relaxed LP problem. 


Theorem 5.3.6 x* is optimal to relaxed problem if and only if it is optimal to the 
original problem. 


Proof Denote by P the feasible region of the original problem and by P’ the 
feasible region of the relaxed problem. Note that both P and P’ are convex sets. 
Necessity is clear. 
Sufficiency. Assume that x* is optimal to the original problem, but not optimal 
to the relaxed problem. According to Theorem 5.3.5, there is solution x’ € P’ such 
that 


oO aa) = 0, (5.21) 


It is clear that x’ ¢ P, and there is some Xx € (x*, x’) that belongs to P. Hence, there 
is some 0 < o < 1 such that 


x—x* =a(x' —x*), (5.22) 

from which and (5.21), it follows that 
cl(x —x*) =ac!(x’ — x*) <0. (5.23) 
According to Lemma 5.3.5, the preceding implies that x* is not optimal to the 


original problem, which is a contradiction, and the claim is therefore valid. Oo 


The preceding says that dropping or adding inactive constraints at an optimal 
solution does not matter with its optimality. In addition, from its proof and 
Definition 2.8.2, we claim the following. 


Corollary 5.3.7 A vector is a feasible descent direction at x* to the relaxed problem 
if and only if so is it to the original problem. 
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5.4 Dual Simplex Algorithm: Tableau Form 


The dual simplex method is very important in optimization, not only because it is 
an efficient method for solving LP problems, but also because it serves as a basic 
tool for solving ILP problems. This section derives its tableau version first. 

Starting from a feasible simplex tableau, in which the right-hand side is 
nonnegative, the tableau simplex algorithm turns negative entries of the objective 
row to nonnegative ones while maintaining non-negativeness of the right-hand side, 
in each iteration. It proceeds, iteration by iteration until all entries of the objective 
row become nonnegative, and then declares an optimal simplex tableau obtained. 

In contrast, the dual simplex method presented in this section starts from the so- 
called dual feasible tableau, in which entries of the objective row are nonnegative, 
and proceeds to achieve non-negativeness of the right-hand side, while maintaining 
non-negativeness of the objective row. 

Consider the standard LP problem (5.1). Let (3.6) be a current dual feasible 
simplex tableau, satisfying Zy > 0 but b # 0. 

This time, we use determine pivot row first by the following rule. 


Rule 5.4.1 (Dual Row Rule) Select row index by 
pe arg min{b; |i =1,...,mb}. 


Thus, the preceding will drop the basic variable x ;,, from basis. 


Lemma 5.4.1 Assume that zy > 0 and bp < 0. If column index set 
J={j¢N|ap; <9} (5.24) 


is empty, then the problem is infeasible. 


Proof zy > 0 indicates dual feasibility. Assume that the dual problem is bounded; 
hence, there is an optimal dual solution. According to Theorem 5.2.5, there is an 
optimal primal solution. Say < > 0, which satisfies the equality, corresponding to 
the pth row of the simplex tableau, i.e., 


£), + ) > Gpj&j = by. 
JEN 


From (5.24) and x > 0, it follows that the left-side of the preceding is nonnegative, 
which contradicts by < 0. Therefore, the problem is dual unbounded, and hence 
infeasible. Oo 


Assume that p has been determined and 258=, +—H 10, 2021 21:06 (5.24)) 
does not hold. Then the following is well-defined. 
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Rule 5.4.2 (Dual Column Rule) Determine B and column index q such that 
B = —2q/4pq = min —2j/Gp j > 0. (5.25) 
JE 


The f is a dual stepsize to be determined. 

Once a pivot is determined, perform elementary transformations to turn Gpg to 1 
and eliminate all other nonzeros in the g-indexed column. Then adding £ times of 
the pth row to the objective row results in a new simplex tableau. It is not difficult 
to show that the new tableau remains dual feasible, that is, its objective row is still 
nonnegative. 

The objective value of the resulting tableau is then 


—~f=-ft+ pb,<-f. (5.26) 


If zy has zero components, it is called dual degenerate. In this case, 6 could vanish, 
and hence, the corresponding objective value remains unchanged. 
Overall steps can be put in the following algorithm (Lemke 1954; Beale 1954). 


Algorithm 5.4.1 (Dual Simplex Algorithm: Tableau Form) Jnitial: a dual fea- 
sible simplex tableau in form (3.6). This algorithm solves the standard LP prob- 
lem (1.8). 


1. Select pivot row index p € arg min{b; |i = 1,...,m)}. 

2. Stop ifb, = 0. 

3. Stop ifJ ={j EN |apj <O}= OH. 

4, Determine pivot column q € arg min jey —Zj/Gp ;- 

5. Convert Gp q to 1, and eliminate the other nonzeros in the column by elementary 


transformations. 
6. Go to step 1. 


Theorem 5.4.2 Under the dual nondegeneracy, Algorithm 5.4.1 terminates either 
at: 


(i) Step 2, achieving a pair of primal and dual optimal solutions; or at 
(ii) Step 3, detecting infeasibility of the problem 


Proof The proof on the termination is similar to the simplex algorithm. The 
meanings of its exits come from Lemmas 3.1.1 and 5.4.1 and discussions preceding 
the algorithm. Oo 


If dual degeneracy presents, the algorithm could stall at the iteration for a time 
in the solution process, even fail to solve a problem due to cycling (Beale 1955). 
Nevertheless, despite the dual degeneracy always occurs, the dual simplex method 
performs satisfactorily in practice, as is just in the primal simplex context. 

Algorithm 5.4.1 starts from a dual feasible simplex tableau. In general, there is a 
need for a dual Phase-I procedure for this purpose (see Sect. 6.1). 
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Example 5.4.1 Solve the following LP problem by Algorithm 5.4.1: 


min f =x; +2x24+ x3 


s.t. 2x7 + x2 + x3 — x4 =1 
—x1 + 4x2 + x3 =2 
x1 + 3x2 <4 


xj 20, j=l,...,4. 
Answer Initial: turn the problem to the standard form by introducing slack 
variables x5,x6 > O in constraints. Then, premultiply the first two constraints by 


—1, respectively: 


min f =x, +2x.+ x3 


s.t. —2x, —x2-—x34+%4 =-1 
x1 — 4x2 — x3 + x5 = —2 
x1 + 3x2 +x =4 


xj>0, j=Hl,...,6, 


which corresponds to an available dual feasible simplex tableau, i.e., 


Iteration 1: 


. min{—1, —2,4}= —-2 <0, p=2. 

. J = {2,3}. 

. min{—2/(—4), —1/(—1)} = 1/2, ¢q =2. 

. Multiply row 2 by —1/4, and then add 1, —3, —2 times of row 2 to rows 1, 3, 4, 
respectively: 


On BWR 


-3/4 1 —I/4 


1/4 -1/4 
~3/4 3/4 
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Iteration 2: 


. min{—1/2, 1/2,5/2} = —1/2 <0, p=1. 

. J = {1,3, 5}. 

_ min{—(3/2)/(—9/4), —(1/2)/(-3/4), —(1/2)/(-1/4)} = 2/3, = 1. 

. Multiply row 1 by —4/9, and then add 1/4; —7/4; —3/2 times of row | to rows 
2, 3, 4, respectively: 


an BW Re 


Xp x2 X3 x4 x5 X6 RHS 
1 1/3 -4/9 1/9 | 2/9 
1 1/3 1/9 2/9 5/9 

-4/3 7/9 5/9 1 | 19/9 

2/3 1/3 —4/3 


The right-hand side of the preceding is now nonnegative, hence obtained an 
optimal simplex tableau. The optimal solution and associated objective value are 


x* = (2/9, 5/9, 0, 0)', f* =4/3. 


5.5 Dual Simplex Algorithm 


In this section, we first derive the revised version of Algorithm 5.4.1 based on the 
equivalence between the simplex tableau and the revised simplex tableau, just as 
what we have done for deriving the simplex method from its tableau version. Then, 
we derive it alternatively to reveal the fact that it essentially solves the dual problem. 

Like the simplex algorithm 3.3.2, in each iteration, the dual simplex algorithm 
computes the objective row, the right-hand side, the pivot column, and row. The 
objective row and/or the right-hand side can be computed in a recurrence manner 
(see (3.14) and (3.15)). The pivot column and row can be computed through Bo} 
and the original data, just as in the simplex method. 

If nonbasic entries in the pivot row are all nonnegative, i.e., 


TA TH -Tpel 
oy =e,N=e,B N=0, 


the dual problem is unbounded; hence, the original problem is infeasible. B~! will 
be updated in the same way as in the simplex method. 

Based on table 3.6.1, therefore, Algorithm 5.4.1 can be revised to the following 
algorithm. 
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Algorithm 5.5.1 (Dual Simplex Algorithm) Initial: (B, N), Bo, zy > 0, XB = 
B~'b, and f = CLR This algorithm solves the standard LP problem (1.8): 


1. Select row index p € argmin{x;, |i =1,...,m}. 
2. Stop if xj, = 0 (optimality achieved). 
3. Compute on = NOB es: 
4. Stop if J ={j|oj <0, j € N} = G (infeasible problem). 
5. Determine B and column index q such that B = —Zq/oq = Minjey —Zj/0;- 
6. Setz;, = B, and update Zy =Zy + Bon, f = f — BX;, if B £0. 
7. Compute ag = Ba, 
8. Update by Xg =XgB — dg, Xq = a, where a = Xj,/Oq- 
9. Update B~' by (3.21). 
10. Update (B, N) by exchanging jy and q. 
11. Go to step 1. 


Alternatively, Algorithm 5.5.1 can be derived by solving the dual problem itself 
as follows. 
Consider the dual problem 


(D) max g = bly 
s.t. Aly <c. 


Given (B, N), B™!. It is easy to verify that y = B~'cg satisfies the dual 
constraints, i.e., 


Zp =cpg — By =0, Zy =cn —N'y >0. (5.27) 


y is a dual feasible basic solution, or geometrically a vertex in the dual feasible 
region 


D={y| A*y <c}. 


In the primal simplex context, y is usually called simplex multipliers. 
Consider the associated primal basic solution 


xp=B'b, xy =0. 
If xg > 0, then x and (y, Z) satisfy the optimality condition and are therefore a pair 
of primal and dual basic optimal solutions. 


Now assume that xg = B~'b # 0. Determine row index p such that 


xj, = min{x;,|i=1,...,m} <0. (5.28) 
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Introduce vector 
= Be, (5.29) 
which with (5.28) gives 
— bth = —b™B Te, = —e}(B-'b) = —X;, > 0, (5.30) 
implying that —/ is an ascent direction, with respect to the dual objective function 


g. 
Now consider the line search scheme below: 


§ = 5 — Bh, (5.31) 


where f is a dual stepsize to be determined. From (5.31), (5.29), and (5.27), it 
follows that 

2p =cg — B'S =cp — B'(5 — Bh) = Bey = 0, (5.32) 

2y =cn —N'$ =cy —N'(y— Bh) = Zw + BNR. (5.33) 

If oy = Nth # 0, then it is seen from (5.33) that a too large 6 > O will lead 

to Zy # 0, as violates the dual feasibility. It is easy to determine the maximum 

possible 6 and the corresponding column index q, subject to Zy > 0 (see step 5 of 

Algorithm 5.4.1 ). Then, drop j, from and enter q to the basis. It is easy to verify 


that 9 is just the dual feasible basic solution, corresponding to the resulting basis. 
The following is valid in the other case. 


Proposition 5.5.1 If oy = N'h > 0, the dual problem is unbounded, and —h is 
an ascent extreme direction in the dual feasible region D. 


Proof If on > 0, it is seen from (5.27) and (5.32), (5.33) that 
Z=c-—A'$>0, VB > 0, 


implying feasibility of the new solution ) given by (5.31). On the other hand, it is 
known from (5.31) and (5.30) that the associated new objective value is 


b'$§ = b'y — px;,, (5.34) 
which goes to oo, as f tends to oo. Thus the dual problem is unbounded. This 
means that —/ is an ascent unbounded direction of D. In fact, it is seen that —h is 
the direction of 1-dimensional face or edge 


fyeR™ | Aly <cya;y = cj, i=1,...,m,iF¢ p} 


and therefore an ascent extreme direction. oO 
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The preceding analysis is actually valid for any given negative x;,. Though far 
from the best, Rule (5.28) is simple and easy to use, and the corresponding objective 
increment is the maximum possible for a unit stepsize. More efficient dual pivot 
rules will be presented in Part II. 

Since the beginning of the nineties of the last century, successful applications of 
the dual steepest-edge rule (Forrest and Goldfarb 1992), some of its approximations 
and bound-flipping (Kirillova et al. 1979) have injected fresh vigor to the dual 
simplex method, so that it becomes one of the most powerful methods for solving 
LP problems (Bixby 2002). 


Example 5.5.1 Solve the following problem by Algorithm 5.5.1: 


min f = x1 + 2x2 + x3 


s.t. —2x, —x2 —x34+x4 =-l 
x1 — 4x2 — x3 + x5 = —2 
Xt + 3x2 +x= 4 


xj=0, f=Hl,...,6 


Answer Initial: B = {4,5, 6}, N = {1,2,3}, B-! =7,zy = (1,2, 1)", xp = 
(-1, —2,4)', f = 0. 


Iteration 1: 


1. minf{f—1, —2,4} = —2 <0, p= 2, xs leaves the basis. 
3. on = (1, —4, -1)". 
5. B = min{2/4, 1/1} = 1/2, q = 2, x2 enters the basis. 
6. Zw = (3/2,0, 1/2)", Z;, = 1/2, f =0- (1/2)(-2) = 1. 
7. Gg = (-1, 4, 3)1. 
8.a = -2/-—4 = 1/2, xg = (-1,-2,4)' - (1/2)(-1,-4,3)' = 
(—1/2, 0, 5/2), % =a = 1/2. 

11/-4 

9 Bli=| i1f/=4 
3/4 1 


10. B = {4,2,6}, N = {1,5,3}, zw = (3/2,1/2,1/2)' > 0, xg = 
(=1/2, 172, 5/21". 


Iteration 2: 


1. min{—1/2, 1/2,5/2} = —1/2 <0, p = 1, x4 leaves the basis. 

3. on = (—9/4, —1/4, —3/4)!. 

5. B = min{(3/2)/(9/4), (1/2)/(1/4), (1/2)/(3/4)} = 2/3, q = 1, x1 enters 
the basis. 

6. Zy = (3/2, 1/2, 1/2)" + (2/3)(—9/4, —1/4, -3/4)" = (0, 1/3, 0)". Z;, = 
2/3, f =1—(2/3)(—1/2) = 4/3. 

7. Gg = (-9/4, -1/4, 7/4)". 
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8. aw = (—1/2)/(—9/4) = 2/9, XB = (—1/2, 1/2, 5/2)" — (2/9)(—9/4, —1/4, 
7/4)? = (0,5/9, 19/9)", x; = 2/9. 


—4/9 11/-4 —4/9 1/9 
9, B= | -1/91 1/-4 | =| -1/9 -2/9 
7/9 1 3/4 1 7/9 5/9 1 


10; BL 1G GN IN: Se ASS): ae Se OTS 1/3, 0)) Se CO rg 
(2/9, 5/9, 19/9)" > 0. 
The optimal solution and objective value: 


x* = (2/9, 5/9,0,0,0,19/9)7,  f* = 4/3. 


5.6 Economic Interpretation of Duality: Shadow Price 


The dual problem is of an interesting economic interpretation. Assume that the 
primal problem 

max f =c!x 

st. Ax<b, x>0 


is a plan model for a manufacturer to produce n products using m resources. The 

available amount of resource i is bj, i = 1,...,m, units; producing a unit of 

product j consumes q;; units of resource i. The profit of a unit of product j is 

cj, j =1,...,n.The goal is to achieve the highest profit with the limited resources. 
The dual problem is 


min bly 
S.t. Aly >c, y>O0. 
Let x* and y* be the primal and dual optimal solutions, respectively. According 


to the strong duality theorem, the associated primal and dual optimal values are 
equal, i.e., 


Therefore, y;* is equal to the increment of the highest profit, created by adding one 
unit of resource i, and can be taken as manufacturer’s assessment for resource i, 
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as is named shadow price by Paul Samuelson.' Shadow price ¥; is the upper price 
limit that the manufacturer can afford to buy resource i. When the market price of 
resource i is lower than shadow price y;*, the manufacturer should consider buying 
it to expand the production scale, whereas he should consider selling it to reduce 
the production scale in the other case. The manufacturer will not buy resource i 
anymore, no matter how low its price is, whenever the optimal solution x* satisfies 
the ith primal inequality constraint slackly, as it implies that resource i is not fully 
used. In fact, the shadow price y* vanishes in this case. 

Let x and y be any primal and dual feasible solutions, respectively; hence, c'x < 
by holds according to the weak duality. The inequality 


clx < bly 


implies that the total profit (output) of the plan is less than the available value (input) 
of the resources. In economic terms, the input-output system is said instable (non- 
optimal) in this case. It is a stable (optimal) system only when the output is equal to 
the input. 

Consider the economic implication of the dual constraints. The manufacturer 
negotiates with the supplier at price y; for resource i, which is calculated to purchase 
resources bj,i = 1,...,m by overall payment by. For j = 1,...,”, on the other 
hand, the supplier asks for resource prices to produce a unit product j being no less 
than the profit of a unit of product /, as it satisfies the jth dual constraint 


m 
) Qi, j Yi = Cj. 


i=1 


If the supplier asks for too high prices, that is, the dual optimal solution y* satisfies 
the jth dual constraint slackly, then x; vanishes, as it implies that the manufacturer 
should not arrange for producing product j at all, no matter how high the profit of a 
unit of the product is. 


5.7 Dual Elimination 


Either the primal or dual simplex algorithm solves the two m x m systems of 
equations in each iteration, which consumes the major cost, especially when the 
number m of the rows of the LP problem is very large. On the other hand, the 
decomposition approach introduced in Chap. 16 of Part IT can deal with large-scale 
LP problems, especially with a very large number n of columns. 


! Paul Samuelson (1915-2009), American economist, the winner of The Nobel Economics Prize 
(1970), the first American to win this prize. 
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In this section, we introduce the so-called dual elimination technique, transform- 
ing the dual problem into an (n — m) x m standard LP problem. Therefore, one 
can solve two (n — m) x (n — m) system in each iteration instead. It is especially 
favorable if m >> n—m. It is interesting that the objective row of the optimal tableau 
of the transformed problem exhibits the primal optimal solution, compared with the 
objective row of the normal optimal tableau, giving the dual optimal solution. 

Consider the tableau of the dual problem (5.2), with max g = b'y replaced by 
min—g = —b'y: 


y z g|RHS 
A' 1 Cc n (5.35) 
—b™ 1 1 


Note that A has full row rank. Through Gauss elimination with row exchanges, the 
preceding can be converted into a tableau in the form below: 


y z g\irhs 

U Gi \q m 
G2 |do n-—m 
x’ il 2 1 


(5.36) 


where U € R”*” is nonsingular upper triangular, G; € R”*", Gz € RUM", 
The preceding corresponds to an equivalent form of the dual problem, i.e., 


min —g = —g + x'z, 
s.t.Goz=d, z>0, (5.37) 
Uy+Giz=d, 


where rank G2 = n — m. Therefore, it is only needed to solve an (n — m) x n 
standard problem with respect to variable z, corresponding to the last n — m + 1 
rows of tableau (5.36), i.e., 


z g | RHS 
G2 do (5.38) 
xt 1| 3 


Starting from the preceding tableau, problem (5.37) (hence the original problem) 
can be solved by a two-phase primal or dual tableau simplex algorithm. In case 
when n — m < m, handling the problem with n — m rows is much cheaper than 
handling the original problem with m rows. When an optimal dual solution z* is 
obtained, solving triangular system 


Uy =d, — Gz" 
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gives the associated solution y*. Consequently, the simplex algorithm will involve 
systems of small order (n — m) x (n — m) rather than the original order m x m. 

It is favorable that the associated primal optimal basic solution is readily 
available. To explain, let us give the following result first. 


Lemma 5.7.1 Jf there is a nonsingular matrix G € R”™*" such that 
GAT = (3): (5.39) 


where U € R”*"™ is nonsingular upper triangular, then the range space of Gs is 
just the null space of A, where G2 consists of the lastn — m rows of G. 


Proof Assume that G; consists of the first m rows of G. It follows from (5.39) that 
(AGT | AGI) = (GT |G) = AGT = (u7|0), (5.40) 


hence leading to AG} = 0. Since G is nonsingular, it holds that rank Gz = n — m. 
Therefore, the range space of G} is the null space of A. oO 


It is not difficult to generalize Lemma 5.7.1 to allow the case when A is not of 
full row rank. In addition, we stress that the lemma is valid for any n x n nonsingular 
matrix G. In particular, performing Gauss elimination with row exchanges on (5.35) 
amounts to premultiplying the first n rows by 


G = GinPn++-GiP\, 


where G; € R”*", i = 1,...,m, are Gauss transformations and P; € R”*", i = 
1,...,m, are permutations. 


Theorem 5.7.2 [f there is an optimal tableau to (5.38), then its bottom row gives a 
basic optimal solution to the original problem. 


Proof Let (5.38) be an optimal simplex tableau. It is known that it is equivalent to 
4 g| RHS 
G2 Goce (5.41) 
b'u-'G, Li bt'U~ ld 
from which it is obtained that 
x=Glu"b. 
Thus, it holds by (5.40) that 


Ax = (AG})U-'b = U'U"b =b. 
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Therefore, the bottom row of (5.36) gives an original solution. By Lemma 5.7.1, in 
addition, the range space of Gs gives the null space of A. This is true for all G's in 
the successive tableaus, generated in the process, and each bottom row comes from 
its predecessor plus a multiple of some row of a Gi. Therefore, the bottom rows 
of all these simplex tableaus give primal solutions. Since the bottom row of the 
optimal simplex tableau is nonnegative and exhibits complementarity with the dual 
basic optimal solution, it gives an optimal basic solution to the original problem. O 


The dual tableau (5.35) is of (2+ 1) x (m+n-+1) order, which is larger than (m+ 
1) x (1 +1), the order of the primal tableau. At a first glance, the approach described 
above seems to be impracticable. It is different if one takes a closer look at it. In fact, 
the triangularization of the former amounts to performing LU factorization of 


Al 
(<) 

(with row exchanges, except for the bottom row). Contents in Sect. 4.3 are still valid 
in this context. When n — m < m, the extra cost is not as much, but the return is 
considerable, since the scale of the linear systems, involved in subsequent iterations, 
is reduced significantly. As for obtaining (5.36), it is only needed to accumulate 
Gauss transformation and permutation factors on (5.35). To obtain y, it is only 
needed to accumulate rows, associated with tableau (5.38), hence only touching 
upon the corresponding components of factors. Alternatively, these factors can be 
stored, and required vectors can be calculated from them and the original data. Note 
that G2 includes (n —m) x (n—m) unit matrix, corresponding to zeros in the bottom 
row. Therefore, there is no need for computing the associated n —m columns. Taking 
the unit matrix as an initial basis, one can get a two-phase primal or dual simplex 
algorithm started directly. 


Example 5.7.1 Solve the following LP problem via the dual elimination: 


min f = 3x1; — x2 + 2x3 —x4+4x%5 — x7 — 2x8, 
s.t. xy +43 — x6 — 2xg = —2, 

—x2 + 4x4 + 2x6 — 3x8 = 9, 

+x3 — x5 + x6 — 2x7 = —S, 

2x, — 5x4 — xg = —18, 

+x2 — 3x5 + x7 = 3, 

—2x, — x3 + 4x5 + 2x6 — 7xg = —13, 

xj>0, j=l,...,8. 
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Answer Construct an initial tableau of form (5.35). 


y1 y2 ¥3 4 ¥5 Yo Zt 22 23 24 25 626 27 


1 2 —2 1 
—2 1 1 
1 1 1 1 
4 = 1 
ll =3 4 1 
= 2 1 2 1 
—2 1 1 
 ) = =] 


Turn the first 6 columns to upper triangular by Gauss elimination. The resulting 
tableau, consisting of the first 6 rows, and the tableau in form (5.38) are as follows, 
respectively: 


yi y2 y3 ya ys Y6 Z1 22 23 z4 25 2% 27 2g] rhs 
1 2 —2 1 3 
—2 1 1 -1 
1 -—2 1 —l 1 -1 
—5 2 0 2 1 3 
-19/5 5 -1 -4/5 1 -2/5 1 6/5 
46/19 25/19 39/19 —6/19 10/19 13/19 1 27/19 
Z1 22 23 24 25 26 27 zg rhs 
—5/2 —5/2 2 -1 —1/2 -1/2 1 —3/2 
761/92 867/92*  —36/23 143/46 289/92 433/92 1 947/92 
—513/92 —723/92 —58/23 63/46 —333/92 —561/92 —1739/92 


There are only 3 rows in the preceding tableau, compared with the original 
problem that involves 7 rows. Therefore, it should be easier to handle. 
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5.8 Bilevel LP: Intercepting Optimal Set 


In the application of LP, there would be a need for the optimal (solution) set or the 
largest optimal face rather than just a single optimal solution, e.g., when one faces 
the following bilevel LP model: 


min fy; = are. 
in: jo =o. x, (5.42) 
s.t. Ax =b, x >0, 


which seeks the optimal set of the first level problem with the objective function f; 
subject to the constraints and then solves the second level problem with the objective 
function f2 over the optimal set. 

By Proposition 2.5.1, it is known that the optimal set exists if so does an 
optimal solution. Further, Lemma 2.5.3 gives the conceptual characterization that 
the optimal set is the intersection of the feasible region and the optimal objective 
isosurface. In this section, we provide an algebraic characterization of it, so that it 
is easy to obtain the optimal set, provided that a dual optimal solution is available. 

Let B = {j1,..., jm} and N = A\B be the optimal basis and nonbasis. Assume 
that the optimal basic solution is 


kg = B'bS0, Gy =O, 
with the optimal value f = ete B, and the associated dual optimal solution is 
Zp=0, Zn =cn—N'y =O, y= Bo" cg. 
Partition the nonbasis N into (N’, N”), such that 
N={jeNn | 2=0}, N" ={j EN | Z > O}. (5.43) 


Using the preceding notation, we claim the following. 


Theorem 5.8.1 The optimal set can be expressed as 
F={xeR" | Bxgp +N xy =b, xp, xy > 0, xyr = 0}. (5.44) 


Proof Let P be the feasible region. 
The LP problem is equivalent to 


min f = f + 2yxn 


5.45 
S.t. xp + B-'Nxy = Xp, XxB,xn > 0. ( ) 
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Its objective function can be written as 
f= ft+Zy,xm, (5.46) 
and the constraint system can be written as 
xp =p — Bo'Nixy, — Bo'Noxny. 
According to Lemma 2.5.3, F is the set of optimal solutions if and only if 

F=PN{xeR" | cx = fh. 
Besides, by (5.46), it holds that 

gray + Zn XN" xeéP. 
From these and the second expression of (5.43), it follows that 


F={xeE€R"|Ax=b, x>0; clx =f} 
={xeR” |xp= Bo!b— Bo N xn, _ BoNoxny, x > 0; ZN, XN = 0} 
= {x €R" | xp = B'b— BN xn, — B-'Noxyy, x = 0; xn, =O} 


= {x ¢R” | xp = B'b— B'Nixn,, xB, xn, = 0; xn, = 0}, 


which is equal to the right-hand side of (5.44). oO 


As the largest optimal face, the set F’s dimension is dependent on the set | NV’| (or 
|N”|) (Proposition 2.2.1). If any index g € N’ is selected to enter the basis, a row 
index p can be determined to leave, just as in the simplex context. For the new basis 
B, the system xg = B-!b—B7!N'x Nn’ renders a new optimal basic solution, which 
is truly new if its predecessor is nondegenerate. This process can be used to obtain 


multiple optimal basic solutions, the number of which is no more than C)") +INI: 


Corollary 5.8.2. A feasible solution is optimal for the LP problem if and only if the 
N” -indexed components of it are all zero. 


Corollary 5.8.3 If the set N’ is empty, there is a unique optimal solution/basis to 
the LP problem. 


The preceding implies that a primal optimal solution is unique if the associated 
dual optimal solution is nondegenerate. 
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Example 5.8.1 Solve the following bilevel LP problem: 


min f) = —3x5 + 5x6 
min ho = —x1 — 4x2 + 2x3 — 6x4 +.x5 + 2x6 
s.t. —3xy — x2 + x3 + 6x4 — 4x6 + x7 — 
5x1 — 2x3 + x4 — x5 + xg = 
2x2 + 3x5 + x6 + x9 = 3 
x20, PS 259. 


Answer The initial feasible simplex tableau is 


-—3 -l 1 6 —4 1 
5 —2 1 —1 1 
2 3* 1 1 
=] =4 2 —6 1 2 
=3 5 


where the bottom and second bottom rows correspond to the first and second 
objective functions, respectively. 


Call Algorithm 3.1.1 to solve the first level problem. 
Iteration 1: 


. min{0, 0, 0, 0, —3,5,0, 0,0} = —3, g =5. 

_L1= {3} 49%. 

. min{3/3}, p = 3. 

. Multiply row 3 by 1/3, and then add 1, —1,3 times of row 3 to rows 2,4,5, 
respectively: 


Nn B&B We 


xX] x2 x3 X4 X5 X6 x7 1X8 
—3 -l 1 6* —4 1 
5 2/3 —2 1 1/3 1 
2/3 1 1/3 
-1 -6 2 —6 1 
2 6 


Iteration 2: 

1. min{0, 2,0, 0,6, 1} => 0. Therefore, the first level problem is minimized with 
optimal value fj = —3 (N’ = {1,3, 4}, N” = {2, 6, 9}). The second objective 
value equals to f2 = 1. Dropping columns corresponding to N” and the bottom 
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row gives the feasible simplex tableau to the second level problem, i.e., 


XxX] X3 X4 X5 X7 
—3 1 6* 1 
5 =2 1 

1 
-1 2 —6 


Call the simplex algorithm 3.1.1. 
Iteration 3: 


. min{—1,2, —6} = —6, q =4. 

.1={1,2} 49%. 

. min{1/6,5/1} = 1/6, p=1. 

. Multiply row 1 by 1/6, and then add —1,6 times of row 1 to rows 2,4, 
respectively: 


On BWR 


Pal x3 X45 x7 
=12 16° «(1 1/6 
11/2* 13/6 —1/6 
1 
=4 3 1 


Iteration 4: 


. min{—4, 3, 1} = —4, q=1. 

.1= {234 90. 

. min{(29/6)/(11/2)}, p = 2. 

. Multiply row 2 by 2/11, and then add 1/2; 4 times of row 2 to rows 1, 4, 
respectively: 


ne We 


x] x3 X4 X5 x7 xg RHS 
—1/33 1 5/33 1/11 | 20/33 
1 = ~-13/33 —1/33, 2/11 | 29/33 
1 1 
47/33 29/33 8/11 | 116/33 
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Iteration 5: 


1. min{47/33, 29/33, 8/11} = 0. 

2. The second level problem is solved with the optimal basic solution x = 
(29/33; 0; 0; 20/33; 1; 0; 0; 0; 0)T, corresponding to the first and second objec- 
tive values f) = —3 and fo = —116/33, respectively. 


5.9 Notes on Duality 


The concept and theorems of duality were first proposed by famous mathematician 
von Neumann. In October 1947, he made foundational discussions on the topic in 
a talk with George B. Dantzig and in a working paper, finished a few weeks later. 
In 1948, Dantzig provided a rigorous proof on the duality theorems in a report. 
Subsequently, (Gale et al. 1951) formulated the duality theorems and proved them 
using Farkas Lemma, independently. Goldman and Tucker (1956a,b) discussed 
theoretical properties of the dual problem systematically. 

As was mentioned, the simplex tableau is just a concise expression of a LP 
problem itself, and all such tableaus created by the primal or dual simplex algorithm 
are equivalent in the sense of their representation of the LP problem. Then the 
following question arises: 

Are the dual problems corresponding to the simplex tableaus equivalent too? 

Consider the dual problem, corresponding to tableau (3.6), i.e., 


max f + b'y’ 
I zi 0 
st. | - ‘+ (°B)a[_ ). Bly =O. 
(ir) CG) . ane 


As their b, N , Zn are not the same, the dual problems corresponding to different 
simplex tableaus are also different. However, such differences are not essential. In 
fact, making variable transformations y’ = B' y — cg, z’ = z and noting 


b=B'b, N=B"'N, Zwy=cyn—N'B ‘cg, f =chBo'b, 
the dual problem can be converted into 
max by, 


BT ZB CB 
s.t. + — ; ' > 0, 
(yr) i) LC aiid 


which is the original dual problem. Therefore, all the generated simplex tableaus 
can be regarded as equivalent. 

In summary, elementary transformations generate equivalent simplex tableaus. 
On the primal side, the right-hand sides give primal basic solutions, and the bottom 
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rows give primal reduced objective functions. On the dual side, the right-hand sides 
render dual reduced objective functions, and the bottom rows dual basic solutions. 
Based on duality, the following is also valid. 


Proposition 5.9.1 Jf it has a dual solution (y, Z), the standard problem (5.1) is 
equivalent to 


max f = y'b+2Z'x, 


5.47 
st. Ax =b, x>0. ( ) 


Proof (¥, Z) satisfies Ay + Z = c, or equivalently, 
cl=ylAtzi, 


Substituting the preceding to the objective of the standard problem and noting the 
constraint system give the objective of (5.47), ie., 


7 = clx = ylAx+Z'x = jlb+z'x, 


and vice versa. oO 


The preceding says that the cost vector c in the standard problem can be replaced 
by any dual solution z, with only a constant difference in objective value. 


References 


Beale EML (1954) An alternative method for linear programming. Proc Camb Philos Soc 
50:513-523 

Beale EML (1955) Cycling in the dual simplex algorithm. Nav Res Logist Q 2:269-275 

Bixby RE (2002) Solving real-world linear problems: a decade and more of progress. Oper Res 
150:3-15 

Forrest JJH, Goldfarb D (1992) Steepest edge simplex algorithm for linear programming. Math 
Program 57:341-374 

Gale D, Kuhn HW, Tucker AW (1951) Linear programming and the theory of games. In: Koopmans 
T (ed) Activity analysis of production and allocation. Wiley, New York, pp 317-329 

Goldman AJ, Tucker AW (1956a) Polyhedral convex cones. In: Kuhn HW, Tucker AW (eds) Linear 
inequalities and related systems. Annals of mathematical studies, vol 38. Princeton University 
Press, Princeton, pp 19-39 

Goldman AJ, Tucker AW (1956b) Theory of linear programming. In: Kuhn HW, Tucker AW (eds) 
Linear inequalities and related systems. Annals of mathematical studies, vol 38. Princeton 
University Press, Princeton, pp 53-97 

Giiler O, Ye Y (1993) Convergence behavior of interior-point algorithms. Math Program 
60:215—228 

Kirillova FM, Gabasov R, Kostyukova OI (1979) A method of solving general linear programming 
problems. Doklady AN BSSR (in Russian) 23:197—200 

Lemke CE (1954) The dual method of solving the linear programming problem. Nav Res Logist 
Q 1:36-47 


Chapter 6 ®) 
Primal-Dual Simplex Method crests 


Methods perform very differently when solving the same problem. It is a common 
case that a problem that is solved slowly by the simplex method would be solved 
fast by the dual simplex method, and vice versa. Consequently, LP packages often 
include multiple options, as it seems to be impossible to predetermine which method 
would be better to solve a given problem. Therefore, scholars have attempted to fuse 
the two methods into one for years. 

In this chapter, we present selected algorithms of both the primal and dual 
features, whose common advantage is Phase-I free, including the mixed two-phase 
simplex algorithm, primal-dual simplex algorithm, self-dual parametric algorithm, 
criss-cross simplex algorithm using the most-obtuse-angle rule, and perturbation 
primal-dual simplex algorithm. In the final section, we make some remarks on the 
criss-cross simplex algorithm. 

Like the simplex algorithm, the finiteness of these algorithms is not guaranteed; 
even the objective value may not change monotonically in the solution process. But 
it can still be expected that cycling rarely happens in practice. 


6.1 Mixed Two-Phase Simplex Algorithm 


It has not yet attracted attention that the dual simplex algorithm itself can be used as 
a Phase-I to get the simplex algorithm started. In fact, if the former terminates, 
it must stop at a feasible basic solution unless detecting the infeasibility of the 
problem. Moreover, from careful examination, we can see that the former not only 
realizes the primal feasibility but also tends to achieve the dual feasibility. The same 
is true with the simplex algorithm as a Phase-I for the dual simplex algorithm. 
Therefore, in principle, there is no need for any other Phase-I procedures, not 
to mention some of them need to introduce the auxiliary function and artificial 
variables. Surprisingly enough, this fact has been ignored by the community for 
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so many years although it is still unknown whether this is a perfect approach 
computationally. 

We demonstrate this approach via examples. 

(A) Dual-primal two-phase scheme: The dual simplex algorithm is used as 
Phase-I for the simplex algorithm. 


Example 6.1.1 Solve the following problem by the dual-primal two-phase scheme: 


min xg = —3x, — 2x2 + 4x3 + 2x4, 

S.t. Xy +2x2 —x3 +2x4 +2x5 = 4, 
2x1 +3x3 —4x4 +X6 = i, 
—x, +2x2 —3x4 t+x7 = -3, 


xj = 0, ji —* reer ee 


Answer Initial tableau is 


xX] x2 x3 X4 X55 Xe x7 f rhs 
1 2 -l 2 1 4 
2 3 -4 1 1 
1 2 3 1 —3 
—-3 -2 4 2 —1 


Phase-I: Call the dual simplex Algorithm 5.4.1. 
Iteration 1: 


. min{4, 1, -3} = -—3 <0, p=3. 

. J = {1,4}. 

. min{—(—3)/(-1), -2/(-3)} = -3, q = 1. 

. Multiply row 3 by —1, and then add —1, —2, 3 times of row 3 to rows 1, 2, 4, 
respectively: 


nN BW ee 


Xx] x2 X3 X4 X5 X6 x7 fe rhs 
4 -1 -l 1 1 1 

4 3 -10 1 2 —5 

1 -2 3 -1 3 
—8 4 11 =3 | —1 9 


Iteration 2: 
1. min{l, —5,3} = —5 <0, p=2. 


4. J = 14), 
4. min{—(11)/(—10)}, q = 4. 
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5. Multiply row 2 by —10, and then add 1, —3, —11 times of row 2 to rows 1, 3, 4, 
respectively: 


x] x2 X3 «2X4 5 X6 xy | ff | rhs 
18/5. —13/10 1 -1/10 4/5 | | 3/2 

—2/5 —3/10 1 —-1/10 —1/5 | | 1/2 

1 —4/5 9/10 3/10 —2/5 3/2 
—18/5 73/10 11/10 —4/5 —1 | 7/2 


The feasibility is achieved. 
Phase-II: Call the simplex Algorithm 3.1.1. 
Iteration 3: 


. min{—18/5, 73/10, 11/10, —4/5} = —18/5 <0, gq =2. 

.1={1, 2} 4G. 

. min{(3/2)/(18/5)} = 5/12, p= 1. 

. Multiply row | by 18/5, then add 2/5, 4/5, 18/5 times of row | to rows 2, 3, 4, 
respectively: 


On BW Re 


xX} x2 X3 «(XG x5 X6 x7 f rhs 
1 —-13/36 5/18 —1/36 2/9 5/12 

—4/9 1 1/9 -1/9 -1/9 2/3 

1 11/18 2/9 5/18 —2/9 11/6 
6 1 1 -1 5 


The reduced costs (6, 1, 1)’ > 0, the optimality is achieved. The optimal basic 
solution and optimal value: 


x* = (11/6, 5/12, 0, 2/3, 0, 0, 0)', fxe=—5. 
(B) Primal-dual two-phase scheme: Phase-I: The primal reduced simplex 


algorithm. 


Example 6.1.2 Solve the same Example 6.1.1 by the primal-dual two-phase 
scheme. 


Answer Phase-I: Call the simplex Algorithm 3.1.1, starting from the first initial 
tableau. 


Iteration 1: 


1. min{—3, —2, 4,2} = -3 <0, g=1. 
2. T=, 46. 
4. min{4/1, 1/2} = 1/2, p =2. 
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5. Multiply row 2 by 2, then add —1,1,3 times of row 2 to rows 1, 3,4, 
respectively: 


Xj x2 x3 X4 X5 X6 x7 f rhs 
2 —5/2 4 1 -1/2 7/2 

1 3/2 —2 1/2 1/2 
2 3/2. —-5 1/2 1 —5/2 

2 17/2. -4 3/2 -1 3/2 


Iteration 2: 


1. {2,,17/2, —4, 3/2} =—-4 <0, g =4. 

3,1 ={l Fd. 

4. min{(7/2)/4} = 7/8, p= 1. 

5. Multiply row | by 4, then add 2, 5, 4 times of row | to rows 2, 3, 4, respectively: 


oal x2 X30 x4 X5 x6 x7 f rhs 
1/2 —5/8 1 1/4 -1/8 7/8 

1 1 1/4 1/2 1/4 9/4 
9/2 —13/8 5/4 -1/8 1 15/8 

6 1 1 -1 5 


The reduced costs (6, 1, 1)? > 0, the dual feasibility is achieved; is so the primal 
feasibility. Therefore, the optimality is achieved without any Phase-II. The optimal 
basic solution obtained is different from the preceding one: 


x* = (9/4, 0,0,7/8,0,0,15/8)', f*=—5. 


6.2 Primal-Dual Simplex Algorithm 


In this section, the primal-dual algorithm (Dantzig et al. 1956) is presented, which is 
an extension of the homonymous algorithm (Ford and Fulkerson 1956) for solving 
transportation problems. 

This algorithm can be regarded as a variant of the dual simplex algorithm. It also 
gets started from a dual feasible solution and proceeds to achieve primal feasibility 
while maintaining dual feasibility and complementarity. Nevertheless, it gets rid of 
artificial variables from the auxiliary problem to fulfill Ax = b while fulfilling the 
sign restriction x > 0, compared with the dual simplex algorithm, which seeks 
toward x > 0 while keeping Ax = b satisfied. 

We are concerned with the standard LP problem (1.8). Let (y, z) be current dual 
feasible solution. 
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To obtain a primal solution matching (y, z), consider the auxiliary prob- 
lem (3.17), written as 


T 


min €=e'Xxq 
6.1 
s.t. Ax +xqg =), X,Xq > 0, eh 
where xq = (Xn+1,---> een is an artificial variable vector. It would be well to 
assume b > 0. Introducing index set 
Q={jeA|z; = 9}, (6.2) 
we define the so-called restricted problem: 
min ¢ = e!x,q 
st. Ax +xg=b, Xq = 0, (6.3) 


xj20, ViedQd, 
xj=0, Vi¢Q. 


Since b > 0, it is clear that the feasible region of the preceding problem is nonempty, 
and hence there is an optimal solution to it. The restricted problem may be viewed 
as one formed by all artificial columns and columns indexed by j belonging to Q. 

Assume that (X, Xq) is an optimal solution to (6.3) with value c , and that w is the 
associated optimal simplex multiplier. 


Theorem 6.2.1 [f the optimal value € vanishes, x and (¥, Z) are a pair of primal 
and dual optimal solutions. 


Proof c = elx, = Oand x, > 0 together imply that x, = 0. Thus, x is a feasible 
solution to the original problem (5.1). By definition of Q, moreover, it holds that 
x'z = 0, exhibiting complementarity. Therefore, x and (¥, Z) are a pair of primal 
and dual optimal solutions. Oo 


When ¢ > 0, otherwise, x could be regarded as the closest to feasibility among 
all those complementary with (y, z). Nevertheless, x is not feasible to the original 
problem because it does not satisfy Ax = b but x > 0 only. In other words, it should 
be possible to improve (y, Zz) by increasing the associated dual objective value. To 
do so, consider the dual problem of (6.3), i.e., 


min b™w 
st. ajw+sj=0, sj 20, jeQ, (6.4) 


wW<e. 


As the simplex multiplier w is just an optimal solution to the preceding problem, it 
follows from the duality that 
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which implies that w is an ascent direction with the dual objective b’ y. This leads 
to the following line search for updating (y, Z): 


j= yt pu, Z=c—A'S. (6.5) 


For being an improved dual feasible solution, it must satisfy the dual constraints for 
some 6 > 0, Le., 


Z=c—Al(y+fw)=Z7+65>0, s=-PA'w. (6.6) 


Since Z > 0, and w satisfies constraints of (6.4), it is known that 


Therefore, if the index set 


J ={j€A\Q|5; =—-a;w <0} (6.7) 
is empty, then (6.6) holds for all 6 > 0, giving a class of dual feasible solutions. 
Since ¢ > 0, the associated dual objective value, i.e., 


b'§ =b'¥ + pb'w =b'y + BE 


tends to +00 as f# increases, which implies dual unboundedness or primal infeasi- 
bility. 

If, otherwise, there is some j € O such that s; = —a 
for the largest possible stepsize 6 such that 


jw < 0, then (6.6) holds 


p=-H=nin |-215 <0, je aol >0. (6.8) 
Sq Spo 


Hence, the resulting dual solution is feasible, corresponding to a strictly larger dual 
objective value, which will be used for the next iteration. 

Let B be the optimal basis of the restricted problem. If a column of B is not 
artificial, it must be indexed by some j € Q such that z; = 0. Since the associated 
reduced cost is zero, i.e., 5; = 0 — aj = 0, it holds that 


2; =z; + Bs; =0, 


implying that 7 also belongs to the next Q. Therefore, the optimal basis of the 
restricted problem can be used as a starting basis for the next iteration. In addition, it 
is seen from (6.8) that there exists at least one index (e.g., g) in 0, which belongs to 
the next Q, and the associated reduced cost is negative, i.e., 5, < 0. In other words, 
there exist new candidates to enter the basis in the next iteration. In each iteration, 
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therefore, the restricted problem can be solved by the primal simplex method. As an 
artificial variable leaves the basis, it is dropped from the original auxiliary problem 
immediately. 

Optimality of the restricted problem is achieved if QM N = &. In case when the 
initial set Q is empty, for instance, all the artificial columns just form an optimal 
basis and the optimal multiplier is w = e; therefore, no simplex steps are needed. 

The steps are summarized into the following. 


Algorithm 6.2.1 (Primal-Dual Simplex Algorithm) Initial: a dual feasible solu- 
tion (y,Z), and associated Q defined by (6.2). B = {n+ 1,...,n +m}, N = 
{1,...,}. This algorithm solves the standard LP problem (1.8). 


1. Carry out the simplex algorithm to solve the restricted auxiliary problem (6.1). 

. Stop if the optimal value of the restricted problem vanishes (optimality achieved). 
. Stop if J defined by 6.7 is empty. (infeasible problem) 

. Compute B by (6.8). 

. Update (¥, Z) by (6.5). 

. Update Q by (6.2) 

. Go to step 1. 


NWOm KR WN 


Although the simplex method is used to solve the restricted problem, any method 
for solving it will apply. The primal-dual simplex method seems to be amenable 
to certain network flow problems because an initial dual feasible solution can be 
obtained easily and the associated restricted problem can be efficiently solved by a 
so-called labeling method (see Papadimitriou and Steiglitz 1982). 

It is noted that the objective value, corresponding to the dual feasible solution, 
increases monotonically iteration by iteration. Therefore, the primal-dual method 
will terminate if each restricted problem encountered is solved within a finite 
number of iterations. 


Example 6.2.1 Solve the following problem by Algorithm 6.2.1: 


min 2x; + 5x2 + x3 + 4x4 4+ 8x5 

s.t. —x, + 4x2 — 2x3 + 2x4 — 6x5 = —1 
xX) + 2x2 + 2x3 —4x5 =8 
x1 + x2 + 2x4+2x5 =2 
820 FST anes, 


Answer Construct the auxiliary problem below: 


min ¢=x6+x%7+ xg 

s.t. xy — 4x2 + 2x3 —2x4+ 6x5 + x5 = 1 
Xx + 2x2 + 2x3 —4x5+x7 =8 
—x1 + x2 + 2x4 + 2x5 + xg =2 
xj 20, j=Hl,...,8. 
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Initial: B = {6,7, 8}, N = {1,...,5}. Since the costs of the original problem are 
positive, a feasible dual solution (y = (0, 0, o0)!, z= (2,5, 1,4, 8)7) is available, 
with O = @. 


Iteration 1: 


1. Since Q = G, no simplex step is needed. 

2. The optimal value of the restricted problem is positive, and the optimal simplex 
multiplier is ® = (1, 1, 1). 

3. 5) = (-1,-4, -47, J = {1,3, 5} 49. 

4. zy = (2,1,8)', 6 = min{2/1, 1/4, 8/4} = 1/4, g =3. 

5. ¥ = (0,0,0)' + 1/4(1, 1, I)" = (1/4, 1/4, 1/4)", 


2 1 7/4 
5 i. 422 6 1/4 21/4 
zw=]1]-[ 122 -4 1/4} =] 0 
4 =f 2 3 1/4 4 
8 7 
6. O = {3}. 


Iteration 2: 
1. Carry out restricted simplex steps of Algorithm 3.3.1: 
subiteration 1: 


(2) Column selection is restricted to QM N = {3}. x3 enters the basis. 
(4) a3 = a3 = (2,2,0)' £0. 
(6) xp = (1,8,2)', a= min{1/2, 8/2} = 1/2, p = 1, x6 leaves the basis, and is 
dropped. 
(7) xp = (1,8, 2)? — 1/2(2, 2, 0) = (0, 7, 2)!. x3 = 1/2. 
1/2 
(8) Bl={ -11 
1 
(9) B= {3,7, 8}, N = {1,2,4, 5}. 


Subiteration 2: 


1/2 0 | 
(1) w=]{ -11 1)/={ 1 J, 
1 1 i 
{ 4230-64" 7-1 - 
cee i 1 }=|_, 
af Be 1 
8 


(2) QNN = {3}N {1, 2,4, 5} = 9. 


2. The optimal value of the restricted problem is positive. 
3. 55 = {-7,-4)', J={2,4 490. 
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4. Zy = (21/4, 4)", B = min{(21/4)/7, 4/4} = 21/28, q =2. 
5. 5 = (1/4, 1/4, 1/4)? + 21/28(-1, 1, 1)? = (1/2, 1, D, 
a i do 6 \" fei es 
: 5 0 
w=[7]-[1 2  -4] [4 =|) 
ft 23 1 
8 13 


6. QO = {3, 2}. 

Iteration 3: 

1. Carry out simplex steps of Algorithm 3.3.1 restricted: 
subiteration 1: 


(2) Column selection is restricted to QM N = {2}. x2 enters the basis. 


1/2 =f ij 
Ce ee \(2)- «| 20 
1 1 1 


(6) xp = (1/2, 7, 2)’, a= min{7/6, 2/1} = 7/6, p = 2, x7 leaves the basis, and 
is dropped. 
(7) XB = (1/2, 7, 2)' — 7/6(—2, 6, 1)? = (17/6, 0, 5/6)", x2 = 7/6. 


1 1/3 72 1/6 1/3 
(8) Bo! -( 1/6 (“ 1 )- (=! 1/6 ) 
—1/61 1 1/6 -1/61 
(9) B = {3,2,8}, N = {1,4, 5}. 


Subiteration 2: 


1/6 1/3 \' /o 1/6 
() w= (=i 1/6 (") = (=i) 
1/6 -1/61 1 1 
1-26\" /1/6 1 
w=-( 1 “ (tis) = (-s2 } 
122 1 ~11/3 


(2) ONN =(3,2}N (1,4, 5} =9. 


N 


. The optimal value of the restricted problem is positive. 

. $y = (—5/3, -11/3)', J = {4,5} 4G. 

. Z7 = (1, 13)", 6 = min{1/(5/3), 13/(11/3)} = 3/5, ¢ = 4. 

. y =(-1/2, 1, 1)? +. 3/5(1/6, —1/6, 1)? = (—2/5, 9/10, 8/5)", 


2 {<6 \ Pas 31/10 
v= (4)-( 1 “ (21) =( 0 ) 
8 9 2 8/5 54/5 


6. O = {3,2,4 


an BW 


“— 
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Iteration 4: 
1. Carry out simplex steps of Algorithm 3.3.1 restricted: 
subiteration 1: 


(2) Column selection is restricted to ON N = {3, 2,4} 1M {1, 4, 5}. x4 enters the 


basis. 
1/6 1/3 —2 -1/3 
(4) ag4= | —1/6 1/6 0 }J= 1/3 £0. 
1/6 -1/61 2 5/3 


(6) XB = (17/6,7/6, 5/6)", « = min{(7/6)/(1/3), (5/6)/(5/3)} = 1/2, p = 
3, xg leaves the basis, and dropped. 


(7) Xp = (17/6, 7/6, 5/6)? — 1/2(—1/3, 1/3, 5/3)? = (3, 1, 0)", x4 = 1/2. 
1 1/5 1/6 1/3 1/5 3/10 1/5 

(8) Bl = —1/5} | -1/6 1/6 =| -1/5 1/5 —1/5 
1 3/5 1/6 —1/61 1/10 —1/10 3/5 


(9) B= {3,2,4}, N = {1,5}. 


2. The optimal value of the restricted problem is zero, optimality achieved. 
The optimal solution and objective value are 


x* = (0, 1,3, 1/2, 0)", f* = 10. 


6.3 Self-Dual Parametric Simplex Algorithm 


This section presents a simplex variant, which solves the standard LP problem 
via the parameterization of the objective function as well as the right-hand side. 
This method is closely related to Orchard-Hays’ work (1954b, 1956) and has been 
used by Smale (1983b) for investigating the worst-case complexity of the simplex 
method. 

Consider the following parametric problem: 


: = Ay 

tn = ee) x (6.9) 
st. Ax =b+0bD, x > 0. 

The standard problem is just the preceding problem with 6 = 0. 

The presented method belongs to a more general homotopy approach. Generating 
a continuous problem deformation, it converts the given problem to a trivially 
solved one and then proceeds backward from the latter to the original by solving all 
problems between them. It can solve the LP problem in a single phase by starting 
from any basis. 
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Assume the availability of a simplex tableau to the LP problem, which is neither 
primal nor dual feasible. It is a simple matter to determine the 6 = 62 > O such that 
the objective row and the right-hand side both become nonnegative after adding 
it properly. This amounts to adding some terms 6c’ and 6b’ respectively to the 
objective function and the right-hand side of the original problem, corresponding to 
6 = 6 = 0. Then, 6 is decreased from 62 down to 0 while maintaining optimality. 
If primal feasibility is violated first in this process, a row index p and a new 62 
are determined; then a column index q is determined by the dual simplex ratio 
test. If otherwise, dual feasibility is violated first, a column index qg and a new 
62 are determined; a row index p is determined by the primal simplex ratio test. 
Subsequent operations are just a normal basis change. 

Assume that the current simplex tableau is optimal to 0 = 6, i.e., 


XB XN RHS 
I N b+b’0, (6.10) 
a+@yio| —f 
The procedure is summarized into the following, where the parametric problem 
with 9 = 0 corresponds to the standard LP problem. 


Algorithm 6.3.1 (Self-Dual Parametric Algorithm: Tableau Form) Given 62 > 
0. Initial: a simplex tableau of the form (6.10), which is optimal with 0 = 62. This 
algorithm solves the standard LP problem. 


1. If Zy < 0, set B = 0; else, determine q and B such that 
a = —24/c, = max{—Z;/Z; |Z; > 0, j € N}. 
hes If b! < 0, seta = 0; else, determine p and a such that 
B = —bp/b, = max{—b;/b; | bj > 0, i =1,...,m}. 


3. Ifa => B, do the following 


(1) Stop with 02 = 0 ifa < 0; (optimality achieved) 
(2) Stop if ag < 0; (unbounded) 
(3) Determine row index p such that 


(bp + B1,0)/Apq = min{ (bj + b}0)/dig | Gig > 0, i=1,...,m), 


where @ is close to 02; 
else 
(4) Stop with 02 = 0 if B < 0; (optimality achieved) 
(5) Stop if J ={j © N | Gp; < 0}; (infeasible) 
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(6) Determine column index q such that 
—(@q + %59)/Gpq = min — (2; + 20) /apj, 


where @ is close to 62. 


4. Ifa > B, set 02 = a else set 02 = B. 

5. Convert apq to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

6. Go to step 1. 


An advantage of the preceding Algorithm is that it solves LP problems in a single 
phase by starting from any basis. It might be said criss-cross because of shuttling 
between primal and dual sides, depending on which of a and £ is larger (see step 
3). Therefore, it seems critical to scale the costs and the right-hand side for an 
equilibrium of their sizes beforehand. On the other hand, the algorithm requires 
more computational effort per iteration, compared with the simplex algorithm. As 
a homotopy algorithm, it seems to be more suitable for solving hard problems. 
Moreover, it stands well as a tool for handling the parametric program (6.9) itself. 

The revised version of it is omitted. 


Example 6.3.1 Solve the following problem by Algorithm 6.3.1: 


min —2x| = 3x2 


s.t. xy + 2x2 + x4 =2 
—2x, —x2. +3 =-1l 
—3x, + 4x0 x5 =-—3 


xj >0, j=1,...,4. 


Answer Put the problem into the following tableau with the costs and the right- 
hand side both parameterized 


x] x2 X3 = -X4 XS RHS 
1 2 1 2 
—2 —1 1 —-1+0 
—3 4* 1 | -3+0 
—2+0 -—3+0 


Given 62 = 4 > 0. 
Iteration 1: 


1. wa = max{—(—2)/1, —(—3)/1} =3, ¢q =2. 

2. 6 = max{—(—1)/1, —(—3)/1} =3, p =3. 

3. a> B. 

(3) min{(—3 + 6)/4} = (-3+ @)/4, p = 3, where 6 is close to 4. 
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4. Set &) = 3. 
5. Taking g = 2, p = 3, the associated basis change leads to 


RHS 
7/2 — 1/20 


-7/4+5/46 


3/4 + 1/46 


—17/4 +7/40 3/4—1/40 | —9/4+ 3/20 — 1/402 


Iteration 2: 


1. @ = max{—(—17/4)/(7/4)} = 17/7, q = 1. 

2. 6 = max{—(—7/4)/(5/4), —(—3/4)/(1/4)} = 3, p = 3. 

. a FZ B. 

min{—(—17/4 + 7/40)/(—3/4))} = —17/3 + 7/30, q = 1, where @ is close 
to 3. 

4. Set 62 = 3 (a degenerate step). 

5. Taking p = 3, gq = 1, then the corresponding basis change leads to 


= 
Dw 
~—" 


x1 x2 X3 X4 x5 RHS 
10/3* 1 1/3 1+ 1/36 
1g 1 =9/3 1+ 1/36 
1 —4/3 -1/3 1— 1/30 
—17/3 +7/30 —2/3+ 1/30 | 2—5/36 + 1/362 


Iteration 3: 


1. @ = max{—(—17/3)/(7/3), —(—2/3)/(1/3)} = 17/7, ¢q = 2. 
2. B = max{—1/(1/3), —1/(1/3)} = -3, p=1. 

3.a> BB. 
(3) min{(1 + 1/30)/(10/3))}, p = 1, where @ is close to 3. 

4. Set 62 = 17/7. 

5. Taking g = 2, p = 1, the associated basis change leads to 


X1 X23 X4 x5 RHS 
1 3/10 1/10* 3/10 + 1/100 
1 11/10 —3/10 21/10 + 7/100 
1 2/5 -1/5 7/5 — 1/50 


17/10—7/108 —1/10+1/100 | 37/10 — 9/56 + 1/1002 


Iteration 4: 


1. a = max{—(—1/10)/(1/10)} = 1, q =5. 
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2. B = max{—(3/10)/(1/10), —(21/10)/(7/10)} = —3, p= 1. 

3. a> B. 

(3) min{(3/10 + 1/106)/(10/3))}, p = 1, where @ is close to 17/7. 

4. Set #2 = 1. 

5. Taking g =5, p = 1 as the pivot, the associated basis change leads to 


xX] x2 x3 X4 N5 RHS 
10 3 1 3+06 
3 1 2 3+06 
1 2 1 2 
1-0 2-0 4— 20 
Iteration 5: 
1. a=0. 
2. B = max{—3/1, —3/1l}=—-3, p= 1. 
3. a> Bp. 


(1) 62 = 0. The optimal basic solution and associated objective value: 


x* = (2,0,3,0,3)), f*=—4, 


6.4 Criss-Cross Algorithm Using Most-Obtuse-Angle Rule 


Zionts (1969) proposed the so-called criss-cross algorithm, which carries out the 
primal and dual simplex iterations alternately. A defect of such doing seems that 
while performing the primal iteration, the reduced costs would already be close to 
nonnegative, but the negative components of the right-hand side are still large in 
magnitude; in other words, the dual iteration should be more favorable to carry out, 
and vice versa. 

The criss-cross algorithm (Yan and Pan 2009) utilizes the most-obtuse-angle 
column and row rules (Part II) in the primal and dual iterations, respectively. 
Therefore, no minimum-ratio test is involved. It does not mechanically perform 
primal and dual iterations alternately, but instead, performs the primal or dual 
iteration according to the current degree of the primal and dual infeasibility. To 
do so, a predetermined threshold value is used to control switching between the two 
sides, so that the primal (dual) iteration is taken if the primal (dual) infeasibility is 
stronger than the dual (primal) infeasibility in a sense. 

A slight variant of the algorithm is formulated as follows. Involving no threshold, 
the variant solves the LP problem in a single phase. 
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Algorithm 6.4.1 (Most-Obtuse-Angle Criss-Cross Algorithm: Tableau Form) 
Initial: the simplex tableau of form (1.8). This algorithm solves the standard LP 
problem. 


1. Determine column index g € argminjey Z;. 

2. Determine row index p € arg min{b; |}i=1,...,mb}. 

3. Compute 4 = min{Z,, bp}. 

4. Stop if ~ > 0 (optimality achieved). 

5. If w = Zq, then: 
(1) Redetermine row index p € argmax{djg |i = 1,..., m}; 
(2) Stop if dpq < 0 (infeasible or unbounded problem); 


else 
(3) Redetermine column index g € argminjen pj; 
(4) Stop if dp, = 0 Cnfeasible problem). 


6. Convert @,, to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
7. Go to step I. 


Note As step 3 involves a comparison between the right-hand side and reduced 
costs, it is important to scale the original data beforehand. If the data are not well- 
scaled, it would be better to set a suitable threshold value t > 0, and use 


w= min{Z,, thy} 


in step 3 instead. 

For simplicity of description, we used the classical prima pivot rule in step 1 and 
the dual pivot rule in step 2 in the preceding Algorithm. In practice, other rules, 
introduced in Part II of this book, should be more favorable. 

The following is the revised version of Algorithm 6.4.1. 


Algorithm 6.4.2 (The Most-Obtuse-Angle Criss-Cross Algorithm) Given t > 
0. initial : (B, N); B7!; ¥g = B~'b, Zy =cy — N'B-' cg and f =cpxpg. This 
algorithm solves the standard LP problem. 


1. Determine g € argminjen Z;. 

2. Determine p € arg min{x;, |i = 1,...,m}. 
3. Compute w = min{x;,, Zg}- 

4. Stop if u > 0 (optimality achieved). 

5. If uw = Zq, then: 


(1) Compute ag = Be ag. 
(2) Stop if ag < 0, (infeasible or unbounded problem). 
(3) Redetermine row index 
p € argmin{x;, | dig = 70, i= 1,...,m}, where 0 = max{djq | i = 
1,...,m}> 0. 
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else 
(4) Compute oy = Ni BT, 
(5) Stop if oy => O Gnfeasible problem). 
(6) Redetermine column index 
q € argmin{Z; | dp; < tO, j © N}, where 6 =minjey ap; < 0. 


7. Update xg = Xp +adg, and set ig =a, f = f —a%y, where w = —Xj,/Apq. 
8. Update zy = zy + Bon, Zip =f, f=f- Bx;,, where B = —Zq/Apq- 
9, Update Bo!. 


10. Update (B, N) by exchanging j, and q. 
11. Go to step 1. 


The computational experiments, reported in Yan and Pan (2009), should well 
reflect the performance of Algorithm 6.4.2 although still preliminary. 

Only the 25 smallest standard NETLIB problems without bounds and ranges 
were tested (Appendix C: Table C.4: Problems AFTRO-SCAGR25). Programmed in 
Fortran without exploiting sparsity, the criss-cross code was compared with MINOS 
5.51 in terms of the number of iterations required: 

The code outperformed MINOS 5.51 with a total iteration ratio of 1.24. 

Since there was no minimum-ratio test involved in each iteration, the CPU time 
margin between it and the simplex algorithm should be even larger. 

Finally, we bring up the following example to demonstrate the solution process. 


Example 6.4.1 Solve the following problem by Algorithm 6.4.1: 


min f => —2x1 = X95 
S.t. xX, —XxX2 +X3 = 2, 
xX) +2x2 +x4 = 8, 
—xX, —Xx2 +x5 = —3, 


xj2>0, jHl,...,5. 


Answer Initial simplex tableau: 


xX] x2 X30°«-X4 XS RHS 
1 —1 1 2 
1 2 1 8 
-1* —-1 1 3 
—2 —1 


Iteration 1: 

1. minf—2, -I} = -2 <0, gq=1. 
2. min{2, 8, —3} = —3 <0, p =3. 
3. w = min{—2, —3} = —3. 
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5. Take p = 3, and carry out the dual iteration. 
5(3) minf—1, -1} = —-l,g=1. 
6. Multiply row 3 by —1, and then add —1, —1, 2 times of row 3 to rows 1,2,4, 
respectively: 


xX] x2 X3 x4 x5 


~ 1 1* 
1 1 1 
1 1 —l 
1 —2 


Iteration 2: 


. min{l, —2}= -—2 <0, g=5. 
. min{f—1,5,3}=—-1 <0, p=1. 
. “w= min{—2,-1} = —2. 
. Take g = 5, and carry out the primal iteration. 
5(1) max{1,1,-l} =1, p=1. 
6. Add —1, 1, 2 times of row | to rows 2,3,4, respectively: 


NWN = 


xX] x2 X3 x4 x5 
—2 1 1 
3% —-1 1 
1 -l 1 
—3 2 


Iteration 3: 


. min{f—3, 2} = -—3 <0, g =2. 
. min{—1,6,2}= —-1 <0, p=1. 
. & = min{—3, —1} = -3. 
. Take g = 2, and carry out the primal iteration. 
5(1) max{—2,3,-1} =3, p= 2. 
6. Multiply row 2 by 1/3, and then add 2; 1;3 times of row 2 to rows 1, 3, 4, 
respectively: 


NAwnd = 


x1 x2 x3 x4 
1/3 2/3 
i ae 


1 27% 1/3 
1 1 
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The preceding simplex tableau is optimal. The basic optimal solution and optimal 
value: 


x= (4, 2,0.0, 3)", f* =—10. 


6.5 Perturbation Primal-Dual Simplex Algorithm 


Although degeneracy hardly causes cycling in practice, it often leads to stalling, 
and degrades the efficiency of the simplex method. The perturbation method aims 
to eliminate this defect (Pan 1999, 2000). 

This method starts from any basis, which is usually neither primal nor dual 
feasible. It performs the normal primal (dual) algorithm but with the negative 
components of each current primal (dual) solution perturbed to some small positive 
numbers. Actually, it solves another problem with its right-hand side b (reduced cost 
c) modified slightly, and finally recovers and solves the original problem with the 
dual (primal) simplex algorithm. If the changes are small enough, there will be no 
need for performing the latter simplex algorithm. 

There are the primal-dual and dual-primal schemes. In this section, we demon- 
strate the former only. The overall steps are put into the following algorithm. 


Algorithm 6.5.1 (Perturbation Simplex Algorithm 1: Tableau Form) Initial: a 
simplex tableau of form (1.8), perturbation parameters 6; > 0, j = 1,...,n. This 
algorithm solves the standard LP problem. 


1. Call the simplex Algorithm 3.1.1: Before each minimum-ratio test, perturb the 
right-hand side: 
bj = 5;,, i€ {i | bj <6;, i=1,...,m). 
2. Stop if returning from step 3 (the problem is lower unbounded). 
3. If returning from step 2, then: 


(1) Reset b = Bob. 
(2) Stop if b>0 (optimality achieved). 
4. Call the dual simplex Algorithm 5.4.1: Before each minimum-ratio test, perturb 
the reduced cost: 
77 =46;, ELI EN |Z < 64)}. 
5. Stop if returning from step 3 (the problem is infeasible). 
6. If returning from step 2, then 


(1) Reset zy = cy — N™B-!cp. 
(2) Stop if zy > 0 (optimality achieved). 


7. Go to step 1. 
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The resetting of the right-hand side in step | is to ensure primal feasibility as 
well as nondegeneracy in primal simplex iterations. At the beginning of step 4, dual 
feasibility is already achieved, and the resetting of the reduced cost is to ensure 
the dual nondegeneracy in dual simplex iterations, as might be suitable for solving 
large-scale and highly degenerate problems. In our preliminary tests, there were few 
switches between primal and dual iterations required. 

The finiteness of the algorithm is not guaranteed since the objective value does 
not strictly monotonically change in the solution process. Based on our experience, 
however, we expect that it is more difficult to construct cycling instances to the 
perturbation algorithm, if possible. There should be a less possibility for stalling, 
and the degeneracy should never be a problem. 

Preliminary computational experiments were done with the preceding Algorithm 
vs the classical two-phase simplex algorithm. There were more than 100 small 
test problems involved, each with no more than tens of inequality constraints and 
decision variables. In the tests, our private classical code requires the iterations about 
2-3 times of those with the code based on Algorithms 6.5.1 (Pan, 1999, 2000). It 
appears that the perturbation algorithm is significantly efficient, compared with the 
classical simplex algorithm, at least for small problems. 

We describe the following simple variant of the preceding algorithm, where only 
the corresponding perturbation parameters are used in the primal (dual) minimum- 
ratio test, whereas the right-hand side (reduced cost) itself is untouched. 


Algorithm 6.5.2 (Perturbation Simplex Algorithm 2: Tableau Form) Initial: a 
simplex tableau of form (1.8), perturbation parameters 6; > 0, j = 1,...,n. This 
algorithm solves the standard LP problem. 


1. Call the simplex Algorithm 3.1.1 but use 4;, instead of b; in the minimum-ratio 
test (3.11) if bj < $j, i=1,...,m. 

2. Stop if returning from step 3. (The problem is lower unbounded.) 

3. Stop if b > 0, if returning from step 2. (optimality achieved) 

4. Call the dual simplex Algorithm 5.4.1 but use 6; instead of z; in the minimum- 
ratio test (5.25) ifzj <6; j EN. 

5. Stop if returning from step 3 (infeasible problem). 

6. Assume returning from step 2. Stop if zy > 0 (optimality achieved). 

7. Go to step 1. 


Example 6.5.1 Solve following problem by Algorithm 6.5.2: 


min f = —2x; —5x2+ 4x3, 


s.t. —2x, +3x2 —5x3 +x4 =-17, 
—4x, +8x2 —2x3 +x5 I 1, 
7x1 —4x2 —x3 +x = —6, 


xj>0, j=l,...,6 
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Answer Set 6; = 1, j =1,...,6. Initial simplex tableau: 


xX] x2 X3 X4 x5 

—2 3 -5 1 

4  g* -2 1 
7 -4  -1 

2 -5 4 


1. Call simplex Algorithm 3.1.1. 
Iteration 1: 


1. minf—2, —5,4} = —5 <0, q =2. 

4. In the minimum-ratio test, by is replaced by 1 (by = -17 < 1): 
min{1/3, 1/8} = 1/8, p = 2. 

5. Multiply row 2 by 1/8, and then add —3, 4, 5 times of row 2 to rows 1,3,4, 


respectively: 
XxX] x2 X3 X4 X5 
—1/2 —17/4 1 —3/8 
—1/2 1 —1/4 1/8 
a* —2 1/2 
—9/2 11/4 5/8 


Iteration 2: 

1. min{—9/2, 11/4, 5/8} = —9/2 <0, q=1. 

4. b3 = —11/2 < 1 is replaced by 1: min{1/5} = 1/5, p =3. 

5. Multiply row 3 by 1/5, and then add 1/2; 1/2; 9/2 times of row 3 to rows 1, 2, 4, 
respectively: 


Xp XxX x3 x4 x5 X6 RHS 
—89/20* 1 —13/40 1/10 | —717/40 
1 —9/20 7/40 1/10 —17/40 
1 —2/5 1/10 1/5 —11/10 
19/20 43/40 9/10 | —173/40 


1. min{19/20, 43/40, 9/10} > 0, 
3. Returned from step 2. 
4. Call dual simplex Algorithm 5.4.1. 


Iteration 3: 


1. min{—717/40, —17/40, —11/10} = —717/40 <0, p=1. 
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4. min{—(19/20)/(—89/20), —(43/40)/(—13/40)} 
= —(19/20)/(—89/20) = 19/89, g = 3. 
5. Multiply row 1 by —20/89, and then add 9/20; 2/5; —19/20 times of row 1 to 
rows 2, 3, 4, respectively: 


x1 x2 X3 X4 N5 X6 ] RHS 
1 20/89 13/178 —2/89 717/178 
1 —9/89 37/178 8/89 247/178 
1 —8/89 23/178 17/89 91/178 


19/89 179/178 82/89 | —1451/178 


Iteration 4: 


1. min{717/178, 247/178, 91/178} => 0. 
6. Returned from step 2. Zy = (19/89, 179/178, 82/89)! > 0. 
The basic optimal solution and optimal value: 


x* = (91/178, 247/178, 717/178, 0, 0, 0)', f* = 1451/178. 


It is noted that the preceding problem was solved by calling the primal and dual 
simplex algorithm once each. 

Although the perturbation parameters 6; were set to 1, it seems to be favorable 
to take different values, no more than 1073. 

We formulate the revised version of Algorithm 6.5.2 as follows. 


Algorithm 6.5.3 (Perturbation Simplex Algorithm 2) Given perturbation param- 
eter 5; > 0, j = 1,...,n. Initial: (B,N), Bo!, xp = Bo'b, ZN = enw — 
N'B-" cg. This algorithm solves the standard LP problem. 


1. Call simplex Algorithm 3.3.1: For the minimum-ratio test, use 6;, in place of x;, 
if 
Xj, < 6;, i= 1,...,m. 

2. Stop if returning from step 5 (infeasible or unbounded problem). 
3. Stop if xg > 0, when returning from step 3. (optimality achieved). 
4. Call the dual simplex Algorithm 3.3.2: For the minimum-ratio test, use 6; in 

place of z; if 

zi < dj, jEN. 

. Stop if returning from step 4 (infeasible problem). 
. Stop if zy => 0, when returning from step 2. (optimality achieved). 
7. Go to step 1. 


nN 
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6.6 Notes on Criss-Cross Simplex Algorithm 


Talacko and Rockafellar (1960), Balinski and Gomory (1963), and Llewellyn (1964) 
first suggested switching between the primal and dual iterations under a certain 
mechanism, but not supported by numerical results. 

Later, Zionts (1969) proposed a so-called criss-cross algorithm, carrying out the 
primal and dual simplex iterations alternately. In the primal iteration, the pivot is 
selected by the primal rule, and the associated basis change is made; in the dual 
iteration, the pivot is selected by the dual rule, then the associated basis change is 
made. In the case when the minimum ratio tends to infinity in the row (column) 
selection, it turns to the dual (primal) iteration, and so on. If no iteration can 
be carried out, it is asserted that there exists no optimal solution; otherwise, if 
primal (dual) feasibility is achieved first, then the primal (dual) simplex algorithm 
is executed subsequently to achieve optimality. Therefore, the algorithm is actually 
a primal or dual Phase-I method, depending on which type of feasibility is attained 
first. Like the standard simplex algorithm, the finiteness of Zionts algorithm is not 
guaranteed. 

A decade later or more, Terlaky (1985), and Wang (1987) proposed a purely com- 
binatorial criss-cross variant independently. Their algorithm performs the primal or 
dual iteration according to the following rule: It determines the minimum among 
indices, associated with all negative components of xg and Zy. If the obtained index 
is associated with zy (xg), then the primal (dual) iteration is carried out. To show 
the finiteness, the algorithm uses a criteria simpler than that of Zionts algorithm to 
detect the nonexistence of an optimal solution. 

However, the performance of all these algorithms cannot compete with the 
simplex algorithm. 
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Chapter 7 
Sensitivity Analysis and Parametric LP od 


After solving an LP problem by the simplex method, there is often the need for 
solving a new problem, yielding from changing some part of the old problem. This 
chapter discusses how to deal with such problems effectively, as changing cost, 
the right-hand side, and coefficient matrix, and adding/dropping variable/constraint, 
and replacing a row/column. This theme, termed sensitivity analysis, has important 
applications in integer and/or mixed-integer linear programming. In addition, we 
introduce the closely related parametric LP, such as parameterizing the costs c 
and/or the right-hand side b. 

The basic strategy used is the so-called warm start. In fact, even for a new 
problem differing significantly from its predecessor, it is often a shortcut to start 
from the final basis obtained. Indeed, it is conceivable that the optimal basis even 
remains unchanged if the original problem changes within a sufficiently small range. 

To begin with, assume that the simplex tableau (3.6) is optimal, associated with 
the pair of primal and dual optimal solutions below: 


Zp =0, Zn=cn—-N'¥>0, J=B cg. (7.1) 


7.1 Change in Cost 


Assume that the cost c is changed to c’. 
For the resulting problem, the primal and dual basic solutions corresponding to 
the basis B are respectively 


/ —1 = / / Ty / —-TV 
Xp=BUb=xgp>0, Zye=cy-Ny, y=B cp. 
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As the primal basic solution does not change at all, primal feasibility remains. If 
zy = 0, therefore, the pair (7.1) is just optimal to the new problem. 

Now assume that zy < 0. In case when changes in costs are slight, corresponding 
changes in the dual basic solution should be tiny too; therefore, it could be expected 
that the number of required iterations for solving the new problem is low if starting 
from B. In particular, changes in nonbasic costs only cause changes in the associated 
reduced costs. Specifically, assume that 


cy = cj + Ac;, JETCN. 


In this case, it is clear that y’ = y, and hence only the following reduced costs need 
to be computed: 


et i ' . 
Zj =; —a;y = 2; + Acj, JeT. 


If they are all nonnegative, then optimality is already achieved; otherwise, the 
simplex algorithm is carried out from this point. 
In some applications, one needs to determine a change range to maintain the 
optimality of B with changes in costs. More precisely, assume that 
ci =c+tAc, (7.2) 
where Ac ¥ 0 is a given vector and T is a scalar to be determined. 
It is clear that B is an optimal basis to the resulting problem if the new reduced 
costs are nonnegative, i.e., 
zy = cn + tAcy — N'B (cp + tAcg) = Zw + tTAzy = O, (7.3) 
where 
Azy = Acy — N'B7Acp. (7.4) 


Introducing 


B = min{—z;/Az; | Azj < 0, j € N}, 
a = min{z;/Az; | Azj > 0, j € N}, 


then the nonnegative values of T, satisfying inequality (7.3), are determined by 
t < 8B, 
while the nonpositive values of T satisfy 


|t| <a. 
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Consequently, it follows that the optimal basis/solution remains optimal whenever 
T satisfies 


—a<t<f. (7.5) 


Example 7.1.1 Find the change range of the cost, corresponding to x1, such that the 
optimal solution remains optimal: 


min f = —4x; — 3x2 — 5x3, 
s.t. 2x, + x2 + 3x3 4+ x5 = 15, 
Xp +X2 +2x3+%X4 = 12, 
—2x1 + x2 — 3x3 4+-x7 = 3, 
2x, +x2 + x6 = 9, 
xj 20, jgol,...,7. 


Answer Example 3.1.1 already gives an optimal simplex tableau to this problem, 
1.e., 


Xp x2 x3 XxX4 x5 X6 x7 |RHS 
: 1 iS 2178 2 
1 —7/12 -1/6 —1/4 1 
1 1/2 1/2 9 
1 —1/4 1/2 —-1/4 


13/6 1/3 1/2 | 37 


The optimal basis and nonbasis are B = {3, 4,2, 1}, N = {5, 6, 7}, corresponding 
to optimal solution and objective value x* = (0, 9, 2, 1,0,0,0)', f* = —37,c* = 
(—4, -3, —5,0,0,0, 0), z\ = (13/6, 1/3, 1/2)°. 

Setting Ac = (1, 0, 0, 0, 0, 0, 0), then Acg = (0,0, 0, 1)7, Acy = (0,0, 0)". 
By (7.4), it holds that 


; 100\° 73-01-24) ° 70 if 

000 iit 4 0 
Aye ho = =| -1/72 
: 001 801-2 0 a 

010 0012 1 


a = min{(13/6)/(1/4), 1/2)/C1/4)} = 2, 6 = min{—(1/3)/(—1/2)} = 2/3. 
By (7.5), the optimal solution remains optimal for 


9 S073. 
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Further, from (7.2) and Ac = (1,0, 0,0, 0,0, 0), it is known that the optimal 
solution remains optimal while the cost c; of x, satisfies 


—4-2=-6<c) < —44 2/3 = —-10/3. 
For instance, when the cost of x; is set to the lower bound —6, it holds by (7.3) that 
zy = (13/6, 1/3, 1/2)" + (—2)(1/4, -1/2, 1/4)" = (5/3, 4/3, 0)" > 0, 


and the optimal simplex tableau is then 


Xy x2 x3 X4 x5 X6 x7 RHS- 
1 1/3, -1/3 2 
1 -7/12  -1/6 —-1/4 1 
1 1/2 1/2 9 
1 -1/4 1/2 -1/4 
5/3 4/3 37 


7.2 Change in Right-Hand Side 


Let the right-hand vector b become 
b'’=b+nAb, (7.6) 


where 7 is a scalar, and Ab is a given vector. 
Corresponding to basis B, the primal and dual basic solutions of the resulting 
problem are 


xp = B\(b+nAb)=xp+nB-!Ab, zy=en—-N 3 =n 20, y= Blep. 


It is clear that the dual solution remains unchanged, so does the dual feasibility. If 
x’, = 0, then the primal solution is also feasible, and hence the two are just a pair 
of primal and dual optimal solutions for the resulting problem. 

Now assume that x, # 0. When |n| is small, the change in x}, is small too, it 
can be expected to solve the resulting problem quickly by the dual simplex method, 
starting from B. In fact, B is an optimal basis for the resulting problem if 7 satisfies 
the following condition: 


Xpt+nv>0, v=B'!Ab, (7.8) 
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The nonpositive values of 7, satisfying the preceding inequality, are determined 
by 


In| <a = min{x;/v; |v; > 0, i = 1,..., mb}, 
while nonnegative values of 7 are determined by 
n < BP = min{—x;/v; | vj, < 0, i=1,...,m}. 


Therefore, (7.7) gives primal and dual basic optimal solutions of the new problem 
whenever 7 satisfies 


—a<n<B. 


Example 7.2.1 Find the change range of the first component of the right-hand side 
such that the optimal basis remains optimal: 


min f = x; +2x2+ x3, 

s.t. —2x, —x2-—x34+x4=-1, 
xy — 4x2 —x34+x5 = —2, 
x1 + 3x2 + x6 = 4, 
xj 20, j=Hl,...,6. 


Answer Example 5.4.1 gives the following optimal simplex tableau to the 
preceding problem: 


Xt X92 X3 x4 X5  X6 RHS 
1 1/3. —4/9 1/9 | 2/9 
1 1/3. -1/9 —2/9 5/9 

—4/3 7/9 5/9 1 | 19/9 

2/3 1/3 | —4/3 


which corresponds to the optimal solution and objective value x* = (2/9, 5/9, 0, 
0,0, 19/9)", f* = 4/3. 

The optimal basis and nonbasis are B = {1, 2,6}, N = {3,4, 5}. 

Set Ab = (1,0, 0)". By the second expression of (7.8), it holds that 


—1 


=0:=1.0 1 —4/9 
v=| 1 —40 o}=|[-1/9], 
f #4 0 7/9 


a = min{(19/9)/(7/9)} = 19/7, 6 = min{—(2/9)/(—4/9), —(/9)/(-1/9)} = 1/2. 
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Therefore, the optimal solution remains optimal whenever 
—19/7<n < 1/2. 


Further, from (7.6) and Ab = (1,0, 05, it is known that the optimal basis remains 
optimal whenever the first component b/ of the right-hand side satisfies 


—1—- (19/7) = —26/7 <b, < -14+-1/2=-1/2. 
If setting b; = —1/2, then it holds by (7.7) that 
xp = (2/9, 5/9, 19/9)" + (1/2)(—4/9, -1/9, 7/9)" = (0, 1/2, 5/2)" > 0. 


The optimal simplex tableau to the new problem is accordingly 


Xp x2 x3 x4 x5 X6 | RHS 
1 1/3 —4/9 1/9 0 
1 1/3 -1/9 —2/9 1/2 
—4/3 7/9 5/9 61 5/2 
2/3 1/3 = 


7.3 Change in Coefficient Matrix 


There are several cases to be discussed separately. 


7.3.1 Dropping Variable 


Assume that the basic variable xp, in row p, is to be dropped. It is clear that if 
the component x, of the optimal solution is equal to zero, deleting it just gives an 
optimal solution to the new problem. 

Now assume that x» > 0. Thus, it holds that 


Consider the nonbasic entries in row p, i.e., 


= T= 1 n 
apj =e,B aj, JEN. 
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Proposition 7.3.1 The new problem is infeasible if 
J'={j EN |Gpj > 0} = 9. 


Proof The feasibility of the new problem implies that the original problem has 
a feasible solution with the pth component vanishing. Assume that the feasible 
solution is > 0, Xp = 0. Then it satisfies the equality, corresponding to row 
p of the optimal tableau of the original problem, i.e., 


Kp + > Ap jXj = bp, 
jen 
the left side of which is less than or equal to 0, but the right side of which is strictly 
greater than 0, as is a contradiction. Therefore, the new problem is infeasible. oO 


If J’ is nonempty, determine column index g such that 
q € argminZ;/dpj, 
jes! * 


where Z;, j € J’ are the reduced costs of the optimal simplex tableau of the original 
problem. So dp, is the pivot. Carry out elementary transformations to drop p from 
and enter qg to the basis. Then deleting the column, corresponding to the nonbasic 
variable x, results in a dual feasible simplex tableau to the new problem. 


Example 7.3.1 Consider the problem 


min f = xj + x2 — 3x3, 

S.t. —2x, — x2 + 4x34 2x5 = —4, 
X1 — 2x2 + x3 +%6=5, (7.9) 
—x, +2x34+ x4 = 3, 
xj>0, jHl,...,6. 


to which there is the optimal simplex tableau below: 


xy x2 x3 x4 x5 X6 ‘| RHS~ 
i =47% =173 2/3 | 13/3 
7 1/3 1/3 | 2/3 

= a) 1 2 
a 4a = sa | HT 


Solve the problem, resulting from (7.9) by dropping variable x1. 
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Answer From the optimal tableau, it is known that x; = 13/3 > 0, J’ = {6} 4 
%, min{(1/3)/(2/3)} = 1/2, q = 6. 

The following dual feasible simplex tableau is obtained by multiplying row 1 by 
3/2, and then adding —1/3, —1/3 times of row | to rows 2,4, respectively: 


x2 X3 x4 X5 XG T RHS 
—2 —1/2 1 13/2 
0 1 1/2 —3/2 
-1 —2 1 |} 2 
1 3/2 | —9/2 


Call dual simplex Algorithm 5.4.1. 
Iteration 1: 


1. min{13/2, —3/2, 2} = —3/2 < 0, p =2. 
3. J = U, detecting dual unboundedness. Therefore, the new problem is infeasible. 


7.3.2 Adding Variable 


Assume that the variable x,+1 is to be added. Let cy41 and a,+1 be the associated 
cost and column, respectively. The new problem is then 


min c!x + Cnt+1Xn415 
St. Ax + 4n41%n41 =b, xX,Xn41 = 0. 


It is clear that B is a feasible basis to the new problem, corresponding to the 


feasible basic solution 
Xnt+ 0 : 


Since the variable x,+1 is nonbasic, the preceding is actually a optimal basic 
solution if associated reduced cost 


= T - 
Zn+1 = Cn+1 — Gn41¥ 


is nonnegative. If this not the case, solve the new problem by the simplex method, 
starting from B. 
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Example 7.3.2 tis known that to problem 


min f = —x, — 2x2 — 4x4 + 3xs, 
s.t. 2x, + x3 = 3, 

Xj +x4 = 2, 

—x2 +x5 = 0, 


xj>0, j=l,...,5, 


there is the optimal simplex tableau below: 


2 1 3 

1 1 2 
—1 1 

3 1 8 


Solve the problem, resulting from the problem by adding variable x6, with ag = 
(1, —2, 3)", c6 = —2. 


Answer The optimal basis for the original problem is B = ({3,4,5}. 
a = Bolag = (1,-2,3)', » = BTcp = (0,-4,3)', % = -2- 
(1, —2, 3)(0, —4, 3) = —19. 

We solve the new problem by the tableau simplex algorithm. The initial simplex 
tableau is obtained by modifying the optimal tableau to the original problem, i.e., 


1 1 =, 2 
-1 1 3% | 
3 1 -19| 8 


Call Algorithm 3.1.1. 
Iteration 1: 


. min{3, 1,-19} = -19 <0, g=6. 

-T={1, 3} 49. 

. min{3/1,0/3} =0, p =3. 

. Multiply row 3 by 1/3, then add —1; 2; 19 times of row 3 to rows 1, 2, 4, 
respectively: 


an BW Re 
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Xx] x2 X30 «OX4 x5 x6 | RHS 

5 1/3* 1 13 3 

{i 1 273 2 
173 1/3 1 

3 -16/3 19/3 8 


Iteration 2: 


. min{3, —16/3, 19/3} = —16/3 <0, gq =2, 

.T={ 9. 

. min{3/(1/3)}, p = 1. 

. Multiply row 1 by 3, then add 2/3; 1/3; 16/3 times of row 1 to rows 2, 3, 4, 
respectively: 


an BWR 


5 2 1 8 
2 1 1 3 
35 16 1 56 


Iteration 3: 


1. min{35, 16, 1} => 0. The optimal basic solution and objective value: 


x* = (0,9,0,8,0,3)', f* =—56. 


7.3.3 Dropping Constraint 


After an LP problem is solved, it is straightforward to find whether or not 
a constraint is active at the optimal solution (equality constraints are active). 
According to Theorem 5.3.6, the optimal basis/solution remains unchanged 
after dropping inactive constraints, and there is nothing needed to be done 
in this case. We might as well assume that some active constraints are 
dropped. 

To solve the new problem, it is advantageous to start with a basis, formed 
by using the old optimal basis as far as possible. We bring up the following to 
demonstrate this principle. 
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Example 7.3.3 Drop the third equality constraint from the problem 


min f = —2x, — x2, 
S.t. xy —x2+x3=2, 
X1 + 2x2 + x4 = 8, 
—x1 —x2+x5 = -3, 
—3x, — 4x2 +. x44+ 3x5 = —-14, 
xj 20, jol,...,5 


and solve the resulting problem. 


Answer The tableau below is from the relaxation problem. 


x1 x2 X30 xX4 
1 -1l 1 
| 2* 1 
—3 -4 1 
—2 -l 


It is known from Example 7.3.5 that the final basis for the original problem is 
B = {5, 2, 1, 3}. Taking {5, 2, 3} as an initial basis for the relaxation problem, we 
convert the tableau to a simplex one by multiplying row 2 by 1/2, and then adding 
1, 4, 1 times of row 2 to rows 1, 3, 4, respectively: 


XxX] x2 X3 X4 X5 
3/2 1 1/2 
1/2 1 1/2 
-1 3 3 
=3/2 12 


x1 x2 x3 X4 x5 

3/2* 1 if 

1/2 1 1/2 
=i 1 1 


—3/2 1/2 
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which is a feasible simplex tableau to the relaxation problem. Call Algorithm 3.1.1. 
Iteration 1: 


. min{—3/2, 1/2} = —3/2, g=1. 

. 2 = {1,2}. 

. min{6/(3/2),4/(1/2)} =4, p=1. 

. Multiply row 1 by 2/3, then add —1/2, 1/3, 3/2 times of row | to rows 2, 3, 4, 
respectively: 


an BW Ee 


xX, x X3 X4 x5 RHS 
1 2/3 1/3 4 
1 -1/3 1/3 2 

2/9 10/9 1 2; 

1 1 10 


Optimality achieved. The optimal basic solution and associate objective value are 


(4 2.000; 3)", fr S=10: 


7.3.4 Adding Constraint 


According to Theorem 5.3.6, adding inactive constraints at an optimal solution does 
not matter with its optimality. As in dropping constraints, nothing needs to do in this 
case either. Now assume that added is an active constraint. 

There are two cases: 

Case 1. Adding an inequality constraint v'x < p such that v'x = ulxp > p. 

Introduce slack variable x, 1 to turn the added inequality to equality. That is, 


T T 
UBXB + UyNXN + Xn+1 = P.- 


Inserting the associated data to the optimal simplex tableau to the old problem as 
row m + 1 gives the following tableau for the new problem: 


XB XN Xntt of RHS 
I N b 
uh UN 1 p 


7.3 Change in Coefficient Matrix 185 


Fori = 1,...,m, add relevant times of row i to row m + 1 to eliminate Te The 
resulting simplex tableau is of form 


XB XN Xn+1 f RHS 

I oN b 
1 pi 
a -1| -f 


(7.10) 


It is noted that only row m+1 changed. Thus, the preceding is a dual feasible tableau 
to the new problem, and can be taken to get Algorithm 5.4.1 started. Note that only 
component 6 = X,+1 of the right-hand side is negative, since the infeasible solution 
to the new problem, i.e., 


(X, Fn41 = 0 — pp <0), 


satisfies the equality associated with row m + 1. 


Example 7.3.4 Solve the problem, generated by adding constraint —2x; + x3 — 
3x4 + 2x6 < —25 to Example 7.3.1. 


Answer Example 7.3.1 gives the optimal basic solution x = (13/3,0,2/ 
3,0, 2, o)T, which does not satisfy the added constraint, because 


—2 x (13/3) + 2/3 = —24/3 > —25. 


Introduce slack variable x7 to the added constraint, and insert a row in accordance 
with the resulting equality to the optimal simplex tableau to the old problem, i.e., 


Pal x2 x3 x4 x5 X6 x7 RHS 
1 4/3 -1/3 2/3 | 13/3 
2/3 1 1/3 1/3 | 2/3 
-1 2 1 |} 2 
= i 23 2 i | =25 
1/3 4/3 1/3 | -7/3 


Multiply row | by 2, then add it to row 4: 

5. Multiply row 2 by —1, then add it to row 4: 

Now the preceding is a dual feasible simplex tableau to the new problem. Call 
dual simplex Algorithm 5.4.1. 
Iteration |: 


1. min{13/3, 2/3, 2, -17}= —17 <0, p=4. 
3. J = 2,440. 
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xy x2 x3 x4 x5 X6 
1 —4/3 —1/3 2/3 
—2/3 1 1/3 1/3 
-1 —2 1 
-8/3 1 —-11/3 10/3 
1/3 4/3 1/3 
Pal x2 X3 X4 x5 X6 
1 —4/3 —1/3 2/3 
—2/3 1 1/3 1/3 
-1 —2 1 
—2* —4 3 
1/3 4/3 1/3 


4. min{—(1/3)/ — 2, —(4/3)/ — 4} = 1/6, gq =2. 
5. Multiply row 4 by —1/2, then add 4/3, 2/3, 1, —1/3 times of row 4 to rows 
1, 2, 3, 5, respectively: 


xX) x2 X3 X4 X5 X6 X7 RHS 
1 7/3 A730 ~ 29/3. |) “47/8 
1 5/3 -2/3 -1/3 | 19/3 

I 2etgye yD ae 

1 2 =3/2. -=172-|) 1772 

2/3 5/6 1/6 | —31/6 


Iteration 2: 


1. min{47/3, 19/3, 21/2, 17/2} > 0. The optimal basic solution and associated 
objective value: 


% = (47/3, 17/2, 19/3, 0, 21/2,0,0)", f= 31/6. 


Case 2. Adding equality constraint v'x = p such that v'x 4 p. 
It might be well to assume 


ult <p. (7.11) 


Consider the auxiliary problem, obtained from adding constraint v'x + 
Xn+1 = p to the old problem. Just as in case (i), a dual feasible simplex tableau, 
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say (7.10), to the auxiliary problem can be obtained. Since x, Xn41 = p— DEX B> 
O satisfies the equality, associated with added row m + 1, it holds that 


pb =Xn41 > 0. (7.12) 


Therefore, the tableau (7.10) is optimal to the auxiliary problem. 


Proposition 7.3.2 Assume that (7.11) holds and that us are nonbasics in rowm+1 
of tableau (7.10). If 


J’ ={jeN |i; >0} 


is empty, then the new problem is infeasible. 


Proof That there is a feasible solution to the new problem implies that there is 
a feasible solution to the auxiliary problem, whose (n + 1)-indexed component 
vanishes. Assume that (x > 0, X,+1 = 0) is such a feasible solution, satisfying 
the equality, associated with row m + 1 of (7.10), i.e., 


Xn+1 + > Vv jx; = p. 
JEN 


Since J’ = @ and Xy, Xn41 > O, the left-hand side of the preceding equality is 

less than or equal to zero. But it is known by (7.12) that the right-hand side of the 

preceding is positive, as a contradiction. Therefore, the new problem is infeasible. 
oO 


Assume now that J’ is nonempty. Select column index g such that 
q € arg min Z;/Am+1 j- 
jes’ 


Taking @m+1q as the pivot, we manipulate tableau (7.10) by elementary transforma- 
tions to drop n + 1 from and enter q to the basis. It is clear that deleting the column, 
associated with x, 1, in the resulting tableau leads to a feasible simplex tableau to 
the new problem. Consequently, the dual simplex method can get itself start from it. 


Example 7.3.5 Itis known that associated to problem 


min f = —2x) — x2, 

s.t. xy —xX2 +43 = 2, 
X} + 2x2 +x4 = 8, 
—x1 — x2 + x5 = —3, 
xj >20, jg=l,...,5 


there is the following optimal tableau: 
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x1 = x2 x3 X4 x5 
1/8 2y3-= 
ir 3". 178 
1 2/3 1/3 
1 1 


Solve the problem, resulting from the preceding problem by adding constraint 
—3x, — 442 +44 4+ 3x5 = —14. 


Answer The optimal solution x = (4, 2, 0, 0, 3yt to the preceding problem does 
not satisfy the added constraint, because 


—3x4-4x24+3x3=-ll>-14. 


Inserting row 4, associated with 3x; + 4x2 — x4 — 3x5 + x6 = 14, to the optimal 
tableau gives the following tableau to the auxiliary problem: 


Xp x2 X3 X4 X5 
1/3 2/3 1 
iL Ss. 18 
1 2/3 1/3 
3. 4 = 33 
1 1 


Executing relevant elementary transformations gives the simplex tableau below, 
corresponding to basis B = {5, 2, 1, 6}: 


xX} x2 x3 x4 x5 
13 2/3 1 
i. 2073? 7/3 
1 2° 478 
1/3* —4/3 
1 1 


J’ = {3} 4 G, min{1/(1/3)} = 3, g = 3. Multiply row 4 by 3, then add 
—1/3; 1/3; —2/3; —1 times of row 4 to rows 1, 2, 3, 5, respectively: 
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xX. x. x3 X4 X5 x6 | RHS 


1 3 —2 —2 
1 —4 3 9 
5 =3 1 


Deleting the column associated with xg from the preceding leads to a dual 
feasible simplex tableau to the new problem, i.e., 


2 1 
1 1 5 
1 3 —2 
1 —-4 9 
e 


Call Algorithm 5.4.1. 
Iteration 1: 


1. min{0,5, —2,9} = —2 <0, p=3. 
3. J = G, detecting infeasibility of the new problem. 


7.3.5 Replacing Row/Column 


In case when a row of the constraint matrix of a problem is replaced by another, the 
resulting problem can be handled via the following two phases: 

The first phase is to solve the problem, obtained by dropping the equality 
constraint corresponding to the row. The second is to solve the new problem, 
obtained by adding the corresponding constraint to the solved problem. It is clear 
that the techniques presented in the previous two sections apply. 

Similarly, a problem, resulting from replacing a column of the constraint matrix, 
can be handled by dropping and adding the columns in two phases. In the rest of 
this section, a direct approach to such a problem will be presented. 

Assume that column ax is replaced by a,. 

If k € N, then the basis B, and hence the related basic solution are not affected, 
but the reduced costs become 


y= — (ay, y=Blcg. 
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If z, = O, it is clear that the optimal solution and basis remain optimal to the new 
problem. 
If, otherwise, k € B, we solve the following auxiliary problem: 


Min X41, 
St. yxy ess, Ae—1XK—1 FOLK + agi Xe ++ + OnXn + agxXn41 = b, 
X,Xn41 = 0. 


(7.13) 


The new problem has a feasible solution if and only if the optimal objective value to 
the auxiliary problem is equal to 0. Moreover, the optimal basis B to the old problem 
is feasible to the auxiliary problem and hence can be taken as an initial basis to get 
the simplex method started. 


Example 7.3.6 From Example 4.5.1, the optimal basis B = {1, 2, 6} is known to 
problem 


min f = x1 + 2x24 x3, 
s.t. —2xj —x2 -—x34+x4=-1, 
xy — 4x2 — x3 + x5 = -2, 
x1 + 3x2 + x6 = 4, 
xj=0, jol,...,6. 
Solve the poe resulting from replacing the first column of the coefficient matrix 
by (1, 1, —2)°. 


Answer The auxiliary problem of form (7.13) has the following tableau (keep the 
original objective row, while positioning the auxiliary objective row at the bottom). 


x] x2 X3° = X4 =X5) XG x7 RHS 
1 -1 -1l 1 —2* -1 
1 -4 -1 1 1 —2 

—2 3 1 1 4 
1 2 1 


Firstly, find the simplex tableau, corresponding to basis B = {7, 2, 6}. 


1. Multiply row 1 by —1/2, then add —1, —1, —1 times of row 1| to rows 2, 3,5, 
respectively: 
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x2 x3 
io 12 

—9/2* —3/2 
5/2 1/2 
2 1 

=12 -172 


X4 X5 = X6 RHS 
—1/2 1/2 
1/2 1 —5/2 
1/2 1 7/2 
1/2 —1/2 
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2. Multiply row 2 by —2/9, then add —1/2, —5/2, —2, 1/2 times of row 2 to rows 
1, 3, 4, 5, respectively: 


x] 
-1/3 
173 
= 75 
5/3 
1/3 


Call Algorithm 3.1.1. 
Iteration 1: 


x2 x3 
1/3* 
1 1/3 
—4/3 
1/3 
=173 


x4 
—4/9 
-1/9 
7/9 
2/9 
4/9 


1/9 


5/9 
4/9 
—1/9 


1. min{1/3, —1/3, 4/9, -1/9} = -1/3 <0, q =3, 


3.2 ={(1,2;4@. 


4. min{(2/9)/(1/3), (5/9)/(1/3)} = 2/3, p= 1. 
5. Multiply row | by 3, then add —1/3, 4/3, —1/3, 1/3 times of row | to rows 
2,3, 4,5, respectively: 


X2 1X3 x4 
1 4/3 
1 1/3 
=| 
2/3 


1/3 
=173 
1 


RHS 
2/3 
1/3 


—4/3 


It is seen that the optimal value of the auxiliary problem vanishes. Thus, deleting 
x7 column and the bottom row gives a feasible simplex tableau to the new problem, 


1.e., 
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XI x2 1X3 x4 x5 X6 RHS 
= 1 4/3 1/3 2/3 
1 1/3 -1/3 1/3 

= af 1 1 3 
2 2/3 1/3 —4/3 


Since the objective row is nonnegative, the preceding is optimal to the new problem, 
with optimal basic solution and objective value as follows: 


x* = (0, 1/3, 2/3,0,0,3)', f* =4/3. 


7.4 Parameterizing Objective Function 


The LP problem involving some parameter is termed parametric LP problem. In 
this section, we consider such type of problem in which a parameter, say @, is in the 
objective function in linear form. 

Given a vector c’, consider the following parametric variant of the standard LP 
problem: 


min f = (c+ 6c)'x, (7.14) 
st. Ax =b, x >0. 
Without loss of generality, we are assumed to solve the preceding problem for 
@ € [0,-+00); the other case 9 € [—o0, 0) can be handled by setting c’ := —c’ 
instead. It is noted that the parameter has nothing to do with feasibility of the 
problem. 
We begin with the following example: 


Example 7.4.1 Solve the following problem for parameter 0 € [0, +00): 


min —(3 + 20)x; — (5 — 0)x2 


s.t. 3x, + 2x2 +x5 =18 
2x2 + x4 = 12 
bal + x3 =4 


420; FS Dicagh 


Answer At first, the problem with t = 0 is solved with Algorithm 3.1.1. The 
following is the optimal simplex tableau, with parameter terms added as the bottom 
row afterward: 
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XX) x2 x3 X4 x5 | RHS 
1 -1/3 1/3 2, 
1 1/2 6 

1 1/3. -1/3 2: 


—20 0 


Eliminate parameter entries, corresponding to basis, by elementary transforma- 
tions to convert the preceding into simplex tableau below: 


XJ X20 4X3 x4 x5 | RHS 
1 -1/3 1/3 2 
1 1/2 6 

1 1/3* —-1/3 2 

3/2 1 36 


7/66 2/30 | —20 


To be optimal to the parametric problem, the nonbasic reduced costs have to be 
nonnegative, i.e., 


%4=3/2-7/60>0, %=1+2/30>0. 


As coefficient 2/3 of @ in the second expression is nonnegative, Z5 maintains 
nonnegative as @ increases from 0. But coefficient —7/6 of @ in the first is negative, 
the increase of 6 is restricted for z4 to be nonnegative. The maximum @ value 
allowed is (3/2)/(7/6) = 9/7. Therefore, the simplex tableau remains optimal 
while 


0 € [0, 9/7], 
corresponding to the same basis B = {1, 2,3} and optimal basic solution x = 
(2; 6;2;0,.0)". 


That z4 < 0 and @ > 9/7 indicates that x4 should enter basis; so take gq = 4. To 
maintain primal feasibility, perform the minimum-ratio test: 


min{2/(1/3),6/(1/2}=6, p=3. 


Thus x3 is determined to leave basis. Then, carrying out the corresponding 
elementary transformations results in 
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x1 x2 


X3 
1 
23/2 
3 
—9/2 
7/20 
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X4 x5 | RHS 
0 4 
1/2* 3 

1 -1 
5/2 27 
—1/20 50 


Now consider the inequalities 73 = —9/2 + 7/26 > 0, z5 = 5/2 — 1/20 > 0. 
The maximum value of @ is only determined by the second inequality, so that is 5. 
Therefore, the tableau maintains optimal for 


6 € (9/7, 5], 


corresponding to the same basis B = {1, 2,4} and optimal basic solution x = 


(4, 3,0, 6, 0)7. 


That z5 < 0 and 6 > 5 suggests entering x5 to basis (g = 5). From minimum- 
ratio test min{3/(1/2)}, p = 2, x2 is determined to leave the basis. Then associated 
elementary transformations lead to 


It is clear that z27 = 


—-5+6 > 0, 73 = 


3 + 26 > O holds for all 


6 > 5, corresponding to optimal basis B = {1, 4,5} and optimal basic solution 
x = (4,0, 0, 12, 6, yr In summary, the results are put in the following table: 


Range of 0 
[0, 9/7] 
(9/7, 5] 
(5, +00) 


Optimal basic solution 


Optimal value 


(2, 6, 2, 0, 0)7 —36 + 20 
(4, 3, 0, 6, 0)T —27 —56 
(4, 0, 0, 12, 6)? —12—80 


From the preceding table, it is seen that there are 3 ranges of 6, each of which 
corresponds to a fixed optimal basic solution and a varying optimal value in linear 
function of 0. It can be shown that the optimal value of the problem is a continuous 
piecewise linear convex function of 9, in general. 
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From the preceding illustration, it is not difficult to form an algorithm for solving 
the problem with parameterized costs. Starting from an optimal basic solution with 
6 = 0, it uses a column rule to clarify the problem’s optimality over the whole 
interval (0, +00), subinterval by subinterval, from the left to the right. 


Algorithm 7.4.1 Initial: (B, N), B~! are optimal for 6 = 6; =0.b = B-'b > 0. 
This algorithm solves the parametric problem (7.14). 

1. Compute Zy = cn — N'y > 0, where § = B~' cp and iy = te = N'y’, 
where y' = Bo Ge 
Stop if zy = 0. (optimal basis with @ € (01, +00)) 
3. Determine q and 62 such that 


N 


02 = —Zq/c, = min{—Z;/Z; |Z; <0, j € N}. 


(optimal basis with 0 € (01, 02]) 

4. Compute ag = Bol aq. 
5. Stop if ag < 0 (unbounded problem with @ € [@1, +00)). 
6. Determine stepsize a and pivot row index p such that 

a = bp/apg = min{b;/a;q | aq > 0; i=1,..., m}. 
Update B~' by (3.21). 
Update (B, N) by exchanging jp and q and set 0, = 62. 
Compute b = B~'b. 
Go to step 1. 


SS AN 


Proofs for finiteness of the preceding algorithm under nondegeneracy and 
meanings of its exits are omitted. 

For a better understanding, take a look at the only parameterized part, that is, the 
objective function 


f=(ct+6c')"x, 


which is a positive combination of c'x and cx. As weight 6 increases, the para- 
metric problem shifts to the problem with objective c’'x and the same constraints. 
For large enough 0, therefore, its optimal basis coincides with the optimal basis of 
the normal problem, if any. If c’ = c, in particular, the optimal basis with 6 = 0 is 
also optimal with 6 € [0, +00), although the corresponding optimal values are not 
constant. 

It is seen that 3 linear systems have to be solved in each iteration of the preceding 
Algorithm, compared with 2 systems involved in simplex contexts. As they share the 
same coefficient matrix, however, systems Bt y = cg and BI y= Cy can be solved 
cheaply. 
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Consider the following problem with parameterized right-hand side: 


min f =c'x 


TAS 
s.t. Ax =b+0bD’, x>0. ( ) 


where b’ € R, and parameter 0 € [0, +00). The other case of 6 € [—oo, 0) can 
be handled by setting b’ = —b’. 
Again, we begin with an example. 


Example 7.5.1 Solve the following problem for parameter 0 € [0, +00): 


min —2x1 = 3x2 


S.t. x1 — x2 + x4 = 6-20 
—x, +402 +.x3 = 
—3x1 + 2x2 xs =-2+06 


xj 20, fH1,...,4 


Answer Put the problem into the following tableau, where a combination of the 
last two columns gives the right-hand side of the parametric problem, i.e., 


xX] X2 X33 X4q  X5 b 6 
1 -l 1 6 —2 

-1 4 1 3 

—3 2. 1 | —2 1 

2 -3 


The problem for 6 = 0 is solved by Algorithm 3.1.1, resulting in optimal simplex 
tableau below: 


Xp X2 x3 Xq  X5 b 6 
1 1/3 4/3 9 —8/3 
1 1/3 1/3 3° =373 

1/3 10/3 1) 19 17/3 

5/3 11/3 27. —22/3 


The increase of 0 should maintain nonnegativity of the right-hand side, such that 


b, = 9 — 8/30 = 0, bz = 3 — 2/30 > 0b3 = 19 — 17/30 = 0. 
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The maximum possible 6 is determined by 


min{—9/(—8/3), —3/(—2/3), -19/(—17/3)} = —19/(—17/3) = 57/17, p =3. 


Therefore, the current basis and solution are optimal for 6 € [0, 57/17]. 

b3 < 0 and @ > 57/17 together suggest that x5 should leave basis, so take 
p = 3. But the nonbasic entries in the third row are all nonnegative, which 
detects dual unboundedness or primal infeasibility of the parametric problem with 
6 € (57/17, +00). 

The results are put to the table below: 


Range of @ Optimal basic solution Optimal value 
(0, 57/17] (9 — 8/30, 3 — 2/30, 0, 0, 19 — 17/36)" —27 + 22/30 
(57/17, +00) infeasible _ 


We formulate the following algorithm for solving the parametric problem. It can 
be viewed as a variant of the dual simplex algorithm; but starting from an optimal 
basis with 0 = 0, it uses a different row rule. 


Algorithm 7.5.1 Initial: (B, N), B~!, zy > 0,b = B-'b > Oandb! = Bo'b’. 
This algorithm solves the parametric problem (7.15). 


1. Stop if b' = 0. (optimal basis with 0 € (61, +00)) 
2. Determine p and 02 such that 


02 = —bp/b), = min{—b;/b; |b; < 0, i=1,...,m}. 


(optimal basis with 0 € (01, 02]) 
3. Compute oy = NIB es, 
4. Stop if J ={j € N | oj; < 0} =Q (infeasible problem with 6 € [61, +00)). 
5. Determine B and column index q such that 
B = —2Z4/0q => minjes —Z;/0;. 
6. Update zn = Zn + Bon, Zj, = B. 
7. Compute ag = Bay. 
8. Update B~' by (3.21). 
9. Update (B, N) by exchanging jp and q. 
0. Compute b = B~'bandb! = B~'Db’, and set 6, = 0». 
1. Goto step 1. 


Discussions can be made similarly to those with Algorithm 7.4.1. For 6 large 
enough, for instance, the parametric problem amounts to minimization of the same 
objective function subject to Ax = b’ (in terms of the same optimal basis, if any). 

Although the previous two Algorithms handle parametric programs by starting 
from the left-hand end, 0, of the interval of 0, it is of course possible, without any 
essential difficulty, to start from the other end of the interval. 


Chapter 8 ®) 
Generalized Simplex Method od 


There are various problems from practice, and all can be put into the following 
general form: 


min f =c!x, 
st. a< Ax <b, (8.1) 
[<x <u: 


where A € R”*", cil,u € R", a,b € R", m <n, rankA = m, anda,b,l,u 
are given vectors. Such types of problems not only have upper and lower bounds on 
variables, but also ranges of Ax. 
Ranges involved in the problems can be eliminated by introducing new variables. 
Setting w = Ax, the preceding problem can be converted to 
min f = c'x, 
st. Ax -—w=0, 
l<x <u, 
a<w<b. 


(8.2) 


Components of x are said to be structural variables, whereas those of w be logical 
variables. 

Therefore, practical LP problems boil down to the following so-called bounded- 
variable problem: 


(8.3) 


where A € R”*" clu € R", b € R”, rank A = m, m < n. Unless indicated 
otherwise, it is assumed that /, uv are finite, and/; < uj. Infinite upper or lower 
bounds can be represented by sufficiently large/small reals. 
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This problem can be converted to the standard form by variable transformations. 
However, such doing increases the problem’s scales. In this chapter, we generalize 
the simplex and dual simplex algorithms to solve the bounded-variable problem 
directly. In addition, we introduce the generalized dual simplex algorithm with 
bound-flipping incorporated. 

In the sequel, the following sign function will be useful: 


1, ift > 0, 
sign(t) = { —1, ifr <0, (8.4) 
0, iff =0. 


Again, assume that the current basis B is as follows: 


B={ji,-.-, jm}, N=A\B. (8.5) 


8.1 Generalized Simplex Algorithm 


Almost all terms for standard LP problems apply to bounded-variable problems. 
Similarly, a solution to Ax = Db is said to be basic if its nonbasic components 
attain one of the associated upper/lower bounds. Therefore, a basic solution is not 
necessarily unique, just as the basic solution in the standard LP context. 

In the sequel, we state results without proof. 


Lemma 8.1.1 Jf there exists a feasible solution to the bounded-variable problem, 
so does a feasible basic solution; if there exists an optimal solution to it, so does an 
optimal basic solution. 


Therefore, it is possible to find an optimal basic solution in the feasible region. 
Let x be a feasible basic solution, associated with B, i.e., 


xj =lj or uj, jen, (8.6) 
lp < Xp = B-'b — B~'NEy < up. (8.7) 


The reduced costs and objective value are then 
Zn =cy—-N'B "cg, f =chB'b+Zyan. (8.8) 
Define the index set 


P={jeNn|x=l)}, M={jen |e =uj). (8.9) 


8.1 Generalized Simplex Algorithm 201 


Then it holds that 
TUN=N, TAN=Z. 


Without confusion, I and I also denote submatrices consisting of corresponding 
columns. 


Lemma 8.1.2 A feasible solution x is optimal if the following set is empty: 
J={jeT |zj <O}U{j ¢T1| Zz; > Of. (8.10) 


Proof Let x’ be any feasible solution. Then it holds that 


Ir is known that 


Hence for any j € N, there are the following two cases: 
(i) j €T. It follows from xi = 1; =; that 
Z jx; > ZjX;5 (8.11) 
Gi) j € 1. From xi < uj = xj, (8.11) follows. Therefore 
pa ee rae 
jen jen 
which implies that 
chB b+ ziat = cl B b+ Zhan. 
The preceding means that the objective value at x’ is no less than that at x, 


therefore x is optimal. 
oO 


Now assume that J is nonempty, and hence a column index q can be determined 
by 


q € argmax |z/;|. 
Jes 
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Assuming g = j;, define the vector 


_ p-l 
ge (") = sianté,) ‘ =), (8.12) 
Axn et—m 


where €g—m is the (n — m)-dimensional unit vector with the (¢ — m)th component 
1. 


Proposition 8.1.1 The vector Ax satisfies 
AAx = 0, clAx > 0. 
Proof It is known by (8.12) that 
AAx = BAxg + NAxn = sign(Zq)(—dqg + dq) = 0. 
From the first formula of (8.8) together with (8.1) and (8.12), it follows that 
—c' Ax = sign(%,)(aj B-'cg — cq) = —sign(2q)2q = U(q)2q = —l2q| < 0. 


(8.13) 
oO 


The preceding Lemma says that —Ax is a descent direction concerning the 
objective cl x. 
Let a > 0 be a stepsize from x. The new iterate is then 
X= xX—aAx. (8.14) 
Since x is feasible, it holds for any a > 0 that 


Ak = Ax —a[B, NJAx = Ax =b. 


The value of stepsize w should enable x to satisfy ] < x < u. Consequently, the 
maximum possible stepsize is 


a = min{u, — 1g, min{a; |i=1,...,mb}}, (8.15) 
where 
(Xj; _ uj,)/AXj;, if AX j < 0, 


aj = } (xj, —1;,)/AX;,, if Ax;, > 0, i=l,...,m. (8.16) 
oO, if Ax;, = 0, 
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There are the following two cases: 


(i) @ = Ug — Ig. In this case, Xg = Ug if Xg = Ig, and Xg = lg if xg = ug. Since 
new solution x is feasible basic, corresponding to the same basis, there is no 
need for any basis change. 

(ii) @ < ug — 1g. Determine row index p ¢€ {1,..., m} such that 


=p. (8.17) 


Then %j, attains its lower bound /;,, or upper bound w;,. In this case, the new 
basis follows from B and N by exchanging j, and q. In addition, it is verified 


that x is a basic solution, corresponding to the new basis. 


By (8.13) and (8.14), it is known that the new objective value is 


A 


f= clk = clk —ac'Ax = clk — alZg| < cl x, 


which strictly decreases if a > 0. The preceding expression leads to the recurrence 
formula of the objective value, i.e., 


hatf = f — alzq\, 


However, we will not use the preceding formula in each iteration, but compute the 
final objective value from original data and the end basis. 


Definition 8.1.1 A feasible solution is degenerate if a basic component of it is on 
one of its bounds. 


Concerning stepsize a, the following two points should be noted: 


(i) In case when a basic solution is degenerate, ~ would vanish, and the basic 
solution remains unchanged even if the basis changes. 

(ii) In practice, the problem should be deemed unbounded if the value of a exceeds 
some predetermined large number. 


From the discussions made above, we make the following conclusions. 


Lemma 8.1.3 Let x be a feasible basic solution. Then the new solution, determined 
by (8.12), (8.14), (8.15) and (8.16), is a basis feasible solution. The corresponding 
objective value does not increase; or strictly decreases if nondegeneracy is assumed. 


The overall steps are put into the following. 


Algorithm 8.1.1 (Generalized Simplex Algorithm) Initial: (B,N),B~! and 
associated feasible basic solution x. This algorithm solves bounded-variable 
problem (8.3). 


1. Compute zy = cy — N'y, where y = Bol cp. 


2. Compute f = c' x, and stop if set J defined by (8.10) is empty. 
3. Select column index g such that g € maxjey |Z;|. 
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4. Compute Axg = —sign(Z,) Bo lag. 

5. Determine stepsize a by (8.15) and (8.16). 

6. Update x by (8.14) and (8.12). 

7. Go to step 1 ifa = ug — 1g; else determine row index p ¢€ {1,..., m} such that 

= Ap. 

8. Update B~! by (3.21). 

9. Update (B, N) by exchanging j, and q. 
10. Go to step 1. 


Theorem 8.1.4 Algorithm 8.1.1 generates a sequence of feasible basic solutions. 
Assuming nondegeneracy throughout the solution process, it terminates at step 2, 
giving an optimal basic solution. 


Proof Its validity comes from Lemmas 8.1.2 and 8.1.3. oO 


Example 8.1.1 Solve the following problem by Algorithm 8.1.1: 


min f = —x; + 3x2, 
s.t. 2< 2x,-3x2 < 10, 
I< x) -x. < 5, 
—xj+2x2 < 0, 
O<x, <6, —2< x. 


Answer Introduce x3, x4, x5 to convert the preceding to 


min f SX 3x2, 


S.t. —2x, +3x2 +23 = 0, 
—xX, +x2 +x4 =0, 
xX] —2x2 +xX5 = 0, 

O<xj <6, —-2<x.<00, 2<2x3<10, IL<xy<5, -wK<x5 <0. 


In the following, the unbounded variables will be handled as —oo or ov, upon which 
only the determination of stepsize touches. 


Initial: B = {3,4,5}, N = {1,2}, Bo! = I, zy = (0.5,-2.5)!, xp = 
(6, 2, —4)', f = 6. The initial solution is feasible basic, where the subscript “(—)” 
labels a component on its lower bound and the superscript “(++)” labels one on its 
upper bound (the same below). 

Iteration 1: 


. y = B-'cgz = (0,0, 0)", Zy = C1, —3)7. 
. J = {2}. 
. maxy |Z;| = 3, q = 2, x2 enters the basis. 
. 22 = Baz = (3, 1, —2)!. 
. & = (6—2)/3 = 4/3,a2 = (2-1)/1=1, 
a3 = (-4—0)/ —2 =2;a = min{o, 4/3, 1,2} = 1. 


AWN Re 


8.1 Generalized Simplex Algorithm 205 


6. Xp = (6,2, —4)' —1 x (3, 1, —2)7 = G, 1, —2)", 
tn = (O(-, -2)' —1 x 0, -1) = OK, -1'. 
7. p = 2, x4 leaves the basis. 
1-3 
8. B= 1 
21 
9, B= (3, 2,5}, N = {1,4}; xp = G, -1, -2)", tn = OK, 11). 


Iteration 2: 


. y = B-Tcg = (0, —3, 0)", Zy = cn — N'y = (1,0)! — (G3, —3)? = (-2, 3)". 
.J={i. 
. max, |Z;| = 2, g = 1, x; enters the basis. 
. a = Ba, = (1, -1,-1)". 
. a =(3—2)/l=1, a2 =(-1—w)/-1=0, 
a3 = (—2—0)/ —1 = 2, a = min{6 — 0, 1, 00, 2} = 1. 
6. xp = (3, -1, -2)' —1 x (1, -1, -1)' = (2,0, -1)', 
xy = (0, 1)?’ —1 x (-1,0) = (1, DT. 
7. p = 1, x3 leaves the basis. 


nABWN 


1 1-3 13 
8. Bli=]11 1 ={1-2 
1 1 yan | gen | 


9. B= (1, 2, 5}, N= {3, 4}; XB SS (1,0, —1)', XN = (2), 15)" 
Iteration 3: 


. y = B-'cg = (-2, 3,0)", Zy =cn — Ny = (0, 0)" — (2, 3)? = (2, -3)7. 
J = {4}. 
. maxy |Z;| = 3,q = 4, x4 enters the basis. 
. 44 = Bo'aq = (—3, -2, -1)'. 
. a, = (1 —6)/-—3 = 5/3, a2 = (0— &)/—2=00, 

a3 = (-1-—0)/-1=1; a =min{5 — 1,5/3,0, I} = 1. 
6. Xp = (1,0, -1)? — 1 x (-3, —2, -1)' = (4,2, 0)", 

Xy = (2,1)' —1x (0,-1) = 2,2)". 

7. p = 3, xs leaves the basis. 


nABWN 


1 -3 1-3 = oe 
8. Bl= 1-2 1-2 =/{-1 -2 
=f ies af cet 


9. B= {12,4}, N 13, 5k xn = G, 2,2), ay = CE5,0™)!. 
Iteration 4: 

. y= B'cz = (1,0, 3)', ZV = cw — Nl y = (0,0)? — 1, 3)? = (1, -3)7. 
et ee fol 


1 
2 
3. maxy |Z;| = 1, g = 3, x3 enters the basis. 
4. 43 = B-1a3 = (—2,-1, -1)!. 
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5. a = (4—6)/-2=1,a2 =(2—~w)/-1l=00, 

a3 = (2—5)/—1=3;a = min{10 — 2, 1, 00, 3} = 1. 
6. Xp = (4,2,2)' —1 x (—2,-1,-1)' = (6, 3, 3), 

Xy = (2,0)' —1 x (-1,0) = (3, 0)!. 
7. p = 1, x1 leaves the basis. 


—1/2 —2 -3 1 3/2 
8. Be | H172 1 a) =e —1/2 
a1fod7 \=1f 21 1 1/2 


9. xg == (3,3,3)', B= {3, 2,4}; xy = (6,0), NW = {1,5}. 
Iteration 5: 


1. y = B-" cg = (0,0, 3/2)", 
Zn =cn — Ny = (1,0)! — (3/2, 3/2)? = (—1/2, —3/2)". 
2. J = W. The optimal basic solution and associated objective value: 


* = (6,3,3,3,0)', f=6-3x3=-3. 


As for the tableau version of Algorithm 8.1.1, the associated simplex tableau is 
the same as the conventional, except there is no need for RHS column to display 
corresponding basic solution. We add three additional rows (u, x, 1), respectively, 
listing upper bounds, variable values, and lower bounds. The simplex tableau is of 
the form below: 


Xp XN 
iN 
ZN 

u ul ul, 
x. | xe. Xe 
1 it 


Based on Table 3.2, Algorithm 8.1.1 can be revised to a tableau form. As dg = 
Boas, (8.16) should be replaced by 


(uj, — X;,)/sign(Zq)di q, if sign(Z,)dig > 0, 
aj = 1 (lj, — x;,)/sign(Zq)dig, if sign(Zq)dig < 0, i=l,...,m. 
oO, if aj, = 0, 
(8.18) 
Algorithm 8.1.2 (Generalized Simplex Algorithm: Tableau Form) Initial: fea- 
sible tableau in form (8.1), associated with x. This algorithm solves the bounded- 
variable problem (8.3). 


1. Compute f = c' x and stop (optimality achieved) if J defined by (8.10) is empty. 
2. Select column index g such that g € maxjey |Z;|. 
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WwW 


. Determine stepsize aw by (8.15), where a; defined by (8.18). 

. Set Xg = —sign(Z,)a and update xg = Xg + asign(Zq)dg ifa #0. 

5. Ifa = ug — lg, go to step 1; else, determine row index p € {1,...,m} such that 
a = Mp. 

6. Convert @pq to | and eliminate the other nonzeros in the column by elementary 
transformations. 

7. Go to step I. 


a 


Note The last 3 rows in the tableau should be updated in each iteration. 


8.1.1 Generalized Phase-I 


Assume that B is the basis at current iteration, associated with basic solution x. 
Introduce the following index sets: 


= {i=1,...,m| xj, <1;,}, 
h={i=l,...,m|x;j, >uj}, 


T={l,...,m}\( UB). 


If J; U fy is empty, then x is feasible. Assume that it is not the case. 
Construct the following auxiliary problem: 


min w= — Deh Xj + eh Xjj> 
s.t. Bxgp =b— Nxy, 
lp<xj<suy, In < xn Suny, 


where the objective function is termed infeasible-sum. Then solution x is feasible to 
the preceding problem. From x, one iteration of Algorithm 8.1.2 can be performed 
with its pivot rule slightly modified. The auxiliary problem is formed iteration by 
iteration with the change of x, until achieving feasibility or detecting infeasibility. 


8.2 Generalized Dual Simplex Algorithm: Tableau Form 


The dual version of the generalized simplex method can be derived using the same 
simplex tableau. Assume that (8.1) is the simplex tableau, associated with B and N 
given by (8.5). 

Let index sets I’ and I be defined by (8.9). Itis known that solution (7g = 0, Zy) 
is dual feasible (as well as complementary with the primal solution) if the following 
condition holds: 


zr>0, zn <0. (8.19) 
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Since components of / and wu are finite, it is always possible to set x, to fulfill the 
preceding condition. The associated xg and objective value are then 


¥p=b—Niky, f =c'x. 


If lg < xp < up also holds, then (x, Z) are a pair of primal and dual optimal basic 
solutions. 
Introduce bound-violation quantity below: 


lj, = X jis If xj; < lis 
Pi = uj, — Xj, If Xj, > uj,, i=1,...,m, (8.20) 
0, Tflj, S Xj Sui, 
and determine row index p by the following rule: 


p € argmax{|p;| |i = 1,..., m}. (8.21) 


It is clear that optimality is achieved if ¢) = 0. Now assume that p, 4 0: then 
Pp > O indicates that x, violates the lower bound while p, < 0 indicates violation 
of the upper bound. Introduce index set 


J={j €T |sign(pp)ap; < O} Uj € M1 | sign(pp)ap; > O}. (8.22) 


It is not difficult to show that the original problem is infeasible if J = @; else, a 
column index qg and 6 can be determined such that 


p= —Z,q/(sign(Pp)apq) = ey —Z;/(sign(Pp)ap;) > 0, (8.23) 


which is the maximum possible stepsize in y-space to maintain dual feasibility (also 
see Sect. 8.3). 

Taking Gp, as pivot, convert the simplex tableau by relevant elementary transfor- 
mations. Then the resulting simplex tableau corresponds to the new basis as follows: 


B = {J1, sees Jp—ls ds Jp+is casts N = N\q U {jp}. 


Let us use (8.1) again to denote the new tableau, and x to denote the associated 
basic solution. Since the new tableau is equivalent to the old, < and x satisfy 


¥p=—Nkn, Xp =—Niy. (8.24) 
Now set the leaving component X;, to the violated bound, i.e., 


£3, =Xj, + Pp, (8.25) 
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and maintain other nonbasic components unchanged, 1.e., 
xXj=xXj, JEN, JF jp. 


Then subtraction of the two equalities in (8.24) gives the following updating 
formula: 


XB = XB = Ppajy- (8.26) 


It is not difficult to show that the new simplex tableau with <x is still dual feasible. 

Noting that z;,, = sign(p,)8 holds for the new tableau, the following recurrence 
formula of the objective value can be derived from X and x, satisfying (8.25), and 
the equality associated with the bottom row of the tableau: 


f=f+2%j,8j, -3),) =f + opzj, =F +B= Ff. 


which means that the objective value increases. 

If all components of zy are nonzero, the simplex tableau is said to be dual 
nondegenerate. In this case, B given by (8.23) is strictly positive, and hence the 
objective value increases strictly. 

Overall steps are put into the following, in which the objective value is calculated 
at the end. 


Algorithm 8.2.1 (Generalized Dual Simplex Algorithm : Tableau Form) Initial: 
dual feasible tableau in form (8.1), associated with x. This algorithm solves the 
bounded-variable problem (8.3). 


. Select row index p by (8.21) together with (8.20). 

. If py = 0, compute f =c"X, and stop (optimality achieved). 

. Stop if J defined by (8.22) is empty (infeasible problem). 

. Determine the column index g by (8.23). 

. Convert dp q to 1, and eliminate other nonzeros in the column by the elementary 
transformations. 

6. Update x by (8.25) and (8.26). 

7. Go to step 1. 


nAbWN Re 


Note The last 3 rows in the simplex tableau should be updated in each iteration. 
The proof of meanings of the algorithm’s exits is delayed to the derivation of its 
revised version. 


Example 8.2.1 Solve the following problem by Algorithm 8.2.1: 


min f = 2x; — x2 + 3x3 — 6x4, 


s.t. —2x, +3x2 —4x3 +244 +25 = 14, 
—3x, +4x2 —5x3 +6x4 +x6 = 16, 
xy —2x2 +2x3 —7x4 +X7 =-15, 


-15 <x, <30, -12<x.<20, -17<x3 < 10, 
—8<x4< 15, -10<x5<26, —13< x6 < 34, 
0<x7< 19. 
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Answer Initial tableau: 


x2 X3 X4 x5 X6 X7 
3 —4 2 1 
4* = 6 1 
= 2 =] 1 
-1 3. -6 


~ as 


Take 


XB = b— Nxy = (—174, —284, 179)"(B = {5,6,7}), f = —191. 


Iteration 1: 


1. p, = —10 — (-174) = 164, po = —13 — (—284) = 271, 
p3 = 19— 179 = —160. max{|164], |271|, | — 160|} = 271 4 0,p = 
2, j2 = 6. 
SJ ={1,2,3, 4 4G. 
- min{—2/(—3), —(—1)/4, —3/(—5), —(—6)/6} = 1/4, ¢ = 2. 
5. Multiply row 2 by 1/4, and then add —3, 2,1 times of row 2 to rows 1,3,4, 
respectively. 
6. x6 = —284+4 271 = —13. 


Xp = (—174, 20, 179)? — 271(—3/4, 1/4, 1/2, —1/4)7. 
= (117/4, —191/4, 87/2)", B = {5, 2, 7}. 


BR WwW 


xX] x2 x3 X4 X5 X6 XT 
1/4 -1/4  —5/2 1 —3/4* 
3/4 1 —5/4* 3/2 1/4 
-1/2 1/2 ° 4 1/2 1 
5/4 7/4 9/2 1/4 
u 30 20 10 15 26 34 19 
x) S15 —191/4 Lt 15 117/4.  —13 87/2 
I 15 12 17 8 10 13 0 
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Iteration 2: 


1. 


RW 


p1 = 26 — 1117/4 = —13/4, pp = —12 — (—191/4) = 143/4, 
3 = 19 — 87/2 = —49/2. max{| — 13/4], |143/4], | — 49/2} = 143/44 
0, 
p=2, jp=2. 


-J={1,3,4 49. 
- min{—(5/4)/(—3/4), —(7/4)/(—5/4), —(—9/2)/(3/2)} = 7/5, q = 3. 
. Multiply row 2 by —4/5, and then add 1/4, 1/2, —7/4 times of row 2 to rows 


1,3,4, respectively. 


. ¥2 = —191/44+ 143/4 = -12, 


Xp = (117/4, -17, 87/2)? — (143/4)(—1/5, —4/5, —2/5)?. 
= (182/5, 58/5, 289/5)', B = {5, 3,7}. 


x2 X3 x4 x5 X6 x7 
-1/5 —14/5 1 —4/5 
—4/5 1 —6/5 —1/5 
—2/5 —23/5* 2/5 1 
7/5 —12/5 3/5 
u 20 10 15 26 34 19 
x —12 58/5 15 182/5 —13 289/5 
I 12 17 -8 —10 —13 0 


Iteration 3: 


1. 


RW 


p1 = 26 — 182/5 = —52/5, po = 10 — 58/5 = —8/5, 
03 = 19 — 289/5 = —194/5. max{| — 52/5], | — 8/5], | — 194/5]} 
= 194/5 #0, p =3, js =7. 


J = {4,6 40. 
. min{—(—12/5)/(—23/5), —(3/5)/(2/5)} = 12/23, q =4. 
. Multiply row 3 by —5/23, and then add 14/5, 6/5, 12/5 times of row 3 to rows 


1,2,4, respectively. 


. X7 = 289/5 — 194/5 = 19. 


¥p = (182/5, 58/5, 15)? — (—194/5)(—14/23, —6/23, —5/23)". 
= (294/23, 34/23, 151/23)7, B = {5, 3, 4}. 


Pal x2 X3 x4 X5 X6 x7 

12/23 1/23 1 —24/23  —14/23 

15/23. —16/23 1 S773. 26/28 

1/23 2/23 1 =9 793 5/23 

7/23. 37/23 9/23 —12/23 
u| 30 20 10 15 26 34 19 
% | -15 eh 34/23 151/23 294/23 -13 19 
1} -15 12 7 =8 <4) -13 0 
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Iteration 4: 
1. 1 = 02 = 03 = 0. The optimal basic solution and optimal value are 


x* = (—15, —12, 34/23, 151/23, 294/23, —13, 19)", 
f* = 2,-1,3, —6)(-15, —12, 34/23, 151/23)? = —1218/23. 


8.2.1 Generalized Dual Phase-I 


A dual Phase-I algorithm is needed to get the generalized dual simplex algorithm 
started. In this aspect, the computational experiments carried out by Koberstein and 
Suhl (2007) are noteworthy. 

To clarify the computational efficiency, they realized several commonly used 
dual Phase-1 algorithms, including the generalization of the dual Phase-I using 
the most-obtuse-angle row rule (named PAN), within MOPS,! and tested with 46 
typical large-scale sparse problems, the maximum among which involves more than 
500,000 constraints and 1,000,000 variables. The computational results showed that 
for most tested problems, PAN required the minimum number of iterations; only 
for a few most difficult problems, the required iterations exceeded an acceptable 
amount. In the latter case, the code turned to the dual infeasibility-sum Phase-I, 
named SDI. Such a combination, PAN+SDI, turned out to be the best among all the 
dual Phase-I algorithms tested and was finally taken as the default option for MOPS’ 
dual simplex algorithm. 

Based on the preceding, the author suggests replacing (8.15) with 


a =min{ug —/,, min{a; | |ajg| = t0, i=1,..., m}}, (8.27) 
6 = max {|a@jg| |i =1,...,m}, 
where 0 < t < l,a;, i=1,...,m, are defined by (8.18). The main idea is to limit 


stepsizes to a certain extent. 
This consideration leads to the following algorithm. 


Algorithm 8.2.2 (Tableau Generalized Dual Phase-I: The Most-Obtuse-Angle 
Rule) Given 0 < t < 1. Initial: simplex tableau in form (8.1) associated with x. 
This algorithm solves the bounded-variable problem (8.3). 


1. If J defined by (8.10) is empty, compute f = c!x, and stop (optimality 
achieved). 

2. Select column index qg such that g € maxjey |Z/|. 

3. Determine stepsize a by (8.27). 

4. Set x, = —sign (Zq) a, and update xg = xg +a sign (Zq) dq ifa £0. 


' MOPS is a package developed by Suhl et al. of College of Production Information Economy and 
Operations Research of Berlin Free University, see Suhl (1994, 1999). 
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5. Ifa = ug — lg, go to step 1; else, determine row index p € {1,...,m} such that 
a = Mp. 

6. Convert a), to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

7. Go to step I. 


8.3 Generalized Dual Simplex Algorithm 


According to Table 3.2, which gives the correspondence between items of the 
simplex and the revised tableau, it is not difficult to formulate the revised version 
of Algorithm 8.2.1. However, we will not do so but derive by dealing with the dual 
bounded-variable problem directly. 

We will use the same symbols as those in the previous section, e.g., primal basic 
solution x, satisfying 


Xs =l,; orus, seEN, 
8.28 
xp = B-'b— B"'Niy, ey 
sets I and II defined by (8.9), dual solution (zg = 0, Zy) satisfying (8.19), and ; 
defined by (8.20), etc. 
Assume that row index p was determined by (8.21). According to Theorem 5.3.6, 
we turn to the relaxed problem by ignoring inactive constraints at x, i.e., 


min f = c' x, 
s.t. Ax = b, 
Ir < xr, (8.29) 
xn Sun, ; 
Li, < Kjys if Pp > 0, 
Xj, < U jn» if Pp < 0. 
Using notation 
1; If 
ee | ioe Hep >, (8.30) 
uj,; If pp <0, 


the relaxed problem can be written 


max bly _ unum + [hwr +N pZjy, 


S.t. Bly +zj,ep = cB, 
ly+ur =cr, 
ly — vy =cq, 


PpZ jp. UM, wr = O. 
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Based on the equality constraints, eliminating vq, wr, and combining (8.9) and 
(8.10) reduce the objective function to 


(b— Mun —Tir)'y +hpzp = chin t+ cpsr + (b — Min — Der)" y + hpz;, 
= cyin + (b— Nin)" y + hpzj,.- (8.31) 


Then setting z-> = wr,Zm = —vm, we transform the relaxed problem to the 
following, equivalently, 


max g(y,z) = cyan + (b— Ny) Ty +fpzj,, 
st. Bly +z ep =cp, 


8.32 
N'y+zy =cn, en) 

Ppzj, 29, zm <0, zr=0O. 

Now, define 
y= B cg, (8.33) 
Zn =cn — N'Y, Zp =0 (8.34) 
and assume that the following condition holds: 

zn <0, zr=0. (8.35) 


Therefore, it is verified that the primal objective value at x and the dual objective 
value at (y, Z) are equal,ie., f = g. 
Using the preceding notation, moreover, the following is valid. 


Lemma 8.3.1 (y,Z) is feasible basic solution to the relaxed problem, which 
exhibits complementarity with x. 


Proof It is clear that (y,Z) is the basic solution to (8.32), satisfying the sign 
constraints at the bottom. So, it is only needed to show 


B'¥+Zj,e€p =cp, N'¥+2Zn =en, (8.36) 
(xp —Ip)'Zp =0, (wn —Xn)'in =0, (8.37) 
(Xjp —lip)Zjp =9, if Pp > 0, (8.38) 
(xj, —Uj,)zj, =90, if pp <0. (8.39) 


From (8.33) and the second expression of (8.34), the first expression of (8.35) 
follows. By the first expression of (8.34), it holds that 


M542 =U ¥+eq —M'F = cq. (8.40) 
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Similarly that 
Ply + Zp =cr. (8.41) 


Therefore, (8.36) is valid. 
By (8.9), on the other hand, it is clear that (8.37) holds; and it is known from the 


second expression of (8.34) that (8.38) or (8.39) holds. oO 
By setting 

vp =0, wg =0, (8.42) 

vn = —Zn, wy = 0, (8.43) 

vp = 0, wr =Zr, (8.44) 


it is not difficult to verify that (y, v, w) is a feasible basic solution to the dual 
problem of (8.3) and exhibits complementarity with x. 


Lemma 8.3.2 [fig < Xp < up holds, then x is an optimal basic solution. 


Proof Since 1p < xg < ug holds, x is clearly a feasible basic solution to the 
original problem (8.3) as well as the relaxed problem (8.29). By Lemma 8.3.1, it 
is known that (y, z) is dual feasible, exhibiting complementarity with x. Therefore, 
the two are primal and dual optimal solutions with respect to the relaxed problem. 
According to Theorem 5.3.6, x is optimal basic solution to (8.3). Oo 


Now we find a new dual solution to improve the objective value. Define search 
direction 


h=-sign(pp) B-'ep, 9;, =sign(pp), (8.45) 
on =—N'h. (8.46) 
Lemma 8.3.3. The preceding search direction satisfies the following conditions: 
B'h+0j,e) =0, N'h+on =0, (8.47) 
(b— N&y)'h+hpoj, > 0. (8.48) 


Proof The first expression is easily verified. We only show (8.48). 
From (8.45) and the second expression of (8.28), it follows that 


h'(b — Nkv) +hpoj, = —sign(pp)e, (Bb — B~' Ny) + sign(pp)hp 


—sign(pp)(e,¥B — hp). 


Then from (8.4) and (8.30), it follows that the right-hand side of the preceding equals 


lin = X jp >0 
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in case of pp» > 0, and equals 


X jp = U jp >0 


in case of pp < 0. oO 


Consider the following line search scheme: 


y=y+Bh, 2, =Z;, + Boj, = sign(op)B, (8.49) 
Zn = Zn + Bon. (8.50) 


Introduce the index set 
J={j¢eT lo; <O}U{j ¢ Io; > O}. (8.51) 


Assume that J 4 §. Then from (8.50) and sign condition Z7q < 0, Zp > 0, it is 
known that the maximum possible stepsize 6 and pivot column index gq satisfy the 
minimum-ratio test 


B = —%q/0q = min —Z;/o; = 0. (8.52) 
je 


If all components of Zy are nonzero, then the solution is dual nondegenerate, hence 
the determined stepsize is positive. 


Lemma 8.3.4 If J 4 G, the new solution, determined by (8.49) and (8.50) together 
with (8.52), is a feasible basic solution to the dual relaxed problem. The respective 
objective value increases, and strictly increases if dual nondegeneracy is assumed. 


Proof From (8.49), the first expression of (8.47) and (8.33), it is known for any 
B => 0 that 


B'$+2),ep = B'¥ + B(B'h+0;),ep) = B'¥ =cp. (8.53) 


From the first expression of (8.49), (8.50), the second expression of (8.47) and 
(8.34), it follows that 


N15 +2y = N19 + BN'h+ Zn + Bon = (N'¥ + Zn) + B(N A + ON) = cen. 
(8.54) 


In addition, by (8.49), (8.50) and (8.47), it is known that rend satisfies the sign 
condition at the bottom of problem (8.32), hence the new solution is feasible basic 
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solution, associated with objective value increasing to 
& = (b— Nky)'$ +hp2;, 
= (b— Nxy)'¥ + B((b— Nky)'h + hpoj,) 
28; (8.55) 
where the inequality comes from (8.48) and 6 > 0. Under dual nondegeneracy, 


B > 0 holds, and hence the strict inequality holds, which means a strict increase in 
the objective value. oO 


When J is empty, (8.47) is not well-defined but the following is valid. 
Lemma 8.3.5 [f J = Y, the original problem (8.3) is infeasible. 
Proof J = % implies that 


on <0, of >0. (8.56) 


Combining the preceding two expressions together with zn < 0 and zr > 0 leads 
to 


Zn =Zn+fhon <0, Zr=Zr+for>0 VB>0. 


Similarly to the proof of Theorem 8.3.4, it can be shown that (¥, Z) satisfies the other 
constraints, with the objective value denoted again by (8.55). Thus, noting (8.48), it 
is known that 


&> OO as Boo. 


Thus, the dual relaxed problem is unbounded, or the relaxed problem is infeasible. 
It is then clear that the original problem is infeasible. Oo 


Now we need to determine a primal solution that is complementary with the dual 
solution. In order to change the value of basic variable xj, from xp to the violated 
bound, it is necessary to change the value of nonbasic variable x, from x, by the 
following range: 


ea ee 
AX =| Pp/ oq! oa (8.57) 
Pp/\Oq\; Xq = Ug. 


Therefore, the new values are 


¥p=Xp—Axgag, Xa =XgtAxq, Xj =Xj, JEN, J #4, (8.58) 
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where dg = Bags associated with the new objective value 
f= ft |Axgiql > f. (8.59) 


Note that all components of xy are the same as those of xy, except for X,. From 
the first expression of (8.58) and the second expression of (8.49), it is known that if 
Pp > O, then Xj, =1;, and Z > O hold, while if pp < 0, then%;, = uj, andZj;, < 0 
hold. Therefore, after updating the basis by exchanging p and q, the x and (5, Z) 
exhibit complementarity, and the latter remains dual feasible. Then, we are ready to 
carry out the next iteration. 

Overall steps are summarized into the following revision of Algorithm 8.2.1. 


Algorithm 8.3.1 (Generalized Dual Simplex Algorithm) Initial: (B, N), Bo!; 
y,z,x satisfying (8.28), (8.33), and (8.34). This algorithm solves the bounded- 
variable problem (8.3). 


1. Select a pivot row index p € arg max{|p;| | i = 1,...,m}, where p; is defined 
by (8.20). 
2. If pp = 0, compute f = c"x, and stop. 
3. Compute oy = —N'h, where h = —sign(pp)B Tey. 
4. Stop if J defined by (8.51) is empty. 
5. Determine #6 and pivot column index q by (8.52). 
6. Compute Ax, by (8.57). 
7. Compute ag = Bolag. 
8. Update x by (8.58). 
9. Update y, zu, Zj, by (8.49) and (8.50). 
10. Update B~! by (3.21). 
11. Update (B, N) by exchanging jp and q. 
12. Go to step I. 


Theorem 8.3.6 Algorithm 8.3.1 generates a sequence of primal and of dual basic 
solutions. Assuming nondegeneracy, it terminates either at 


(i) Step 2, giving a pair of primal and dual basic optimal solutions; or at 
(ii) Step 4, detecting the infeasibility of the problem. 


Proof The validity comes from Lemmas 8.3.2, 8.3.4, and 8.3.5, and related 
discussions, made preceding Algorithm 8.3.1. oO 


Example 8.3.1 Solve the following problem by Algorithm 8.3.1: 


min f = x; + 2x2 — 2x3, 


S.t. —2x, +x. +234 24 = 0, 
—xX, —x2 +3 + x5 = 0, 
xX} —X2 —2x3 +x56 =0, 

l<x, <5, -2<%1. <0, —354%350, 2< x4 55, 


O<x5 <6, -3<x6 <0. 


8.3. Generalized Dual Simplex Algorithm 219 


Answer Initial: B = {4,5,6}, N = {1,2,3}, Bl = J, ky _ 
(1), —2-), 01, xp = (4, -1, -3)7, ¥ = (0,0,0), zy = (1,2, -2)', f = 
—3. 


Iteration 1: 


. max{0, |O — (—1)|, 0} = 1, p = 2, x5 leaves the basis. 
. h =—sign(p2)B-Te2 = (0,-1,0)', oy =—Nth = (-1,-1,1)'. 
. J ={1,2,3} 490. 
. B=min{—-1/(-1), —2/(-1), -(-2)/=1, gH. 
. Ax, = p2/|o1| = 1. 
a = B~'a, = (-2,-1, 1)". 
. Xp = Xp — Axia = (4, —-1, —3)' — (-2, -1, 1)? = (6,0, —4)'. 
xy =Xy + Axyje; = (1, —2,0)' + (1, 0,0)" = (2, —2, 0)". 
9. y=y+ Bh =(0,—-1,0)'4+1 x (0, —-1, 0)? = (0, —2, 0)", 
Zn = Zn + Bon = (1,2, — arn era (0,1, —D?, 
Z5 = sign(p2)B = 1. 
19, 
10. Update Bo! = ee 


ll. xp = (6,2, aa B = {4,1,6 3y = O.),-25,07)' zy = 
(1,1,-1)7, 
N = {5, 2, 3}. 

Iteration 2: 

1. max{|5 — 6|, 0, (—3) — (—4)} = 1, p = 1, x4 leaves the basis. 

3. h = —sign(p,)B-Te, = (1, —2, 0)', on = —N*hA = (2, —3, LT. 
4. J = {2,3} 49. 
5. B =min{—1/(—3), —(—1)/1} = 1/3, g =2. 
6 
7 
8 


. Ax2 = —pi/|o2| = 1/3. 

. G2 = Baz = (B, 1, -2)". 

. ¥B = (6, 2, —4)? — (1/3), 1, —2)? = (5, 5/3, —10/3)', 
Xv = Xn + Axrer = (0, —2, 0)7 + (0, 1/3, 0)' = (0, —5/3, 0)". 

9. y= H+ Bh= (0, -2,0)', 
zn = Zw + Bon = (1, 1,-1)7 + (1/3), -3, 1)? = (5/3, 0, —2/3)", 
Z4 = sign(p1)B = —1/3. 


1/3 p28 1/3 —2/3 
10. UpdateB-! = | -1/31 ai. PSHE esigs. li 
2/3 1 11 2/3 —1/31 
11. B= {2,1,6}, N = (5,4, 3}, #8 = (—5/3, 5/3, —10/3)", 


Xn = (O-), 54), OM)T, 
zw = (5/3, -1/3, —2/3)'. 
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Iteration 3: 


1. max{0, 0, |(—3) — (—10/3)|} = 1/3, p = 3, x6 leaves the basis. 
3. h = —sign(p3)B-'e3 = (—2/3, 1/3, —-1)', on = —NTh = (1/3, 2/3, 
—5/3)!. 
J ={1, 2348. 
2 = q € min{—(5/3)/(—1/3), —(—1/3)/(2/3)}, B = 1/2, x4 enters the basis. 
. Ax2 = p3/|o2| = —(1/3)/(2/3) = -1/2. 
a2 = B-la = (1/3, —1/3, 2/3)". 
. ¥B = (—5/3, 5/3, —10/3)" — (—1/2)(1/3, —1/3, 2/3)? = (—3/2, 3/2, —3)!, 
ky = Xy + Axre2 = (0,5, 0)" + (0, —1/2, 0)? = (0, 9/2, 0)T. 
9. y = (0, —2, 0)! + (1/2)(—2/3, 1/3, -1)? = (—1/3, —11/6, —1/2)", 
Zn = Zn + Bon = (5/3, —1/3, —2/3)? + (1/2)(—1/3, 2/3, —5/3)? 
= (3/2, 0, —3/2)', 
Z = sign(p3)B = 1/2. 


ONNDAR 


i 12) (i328 21/2 =1/2 
10. UpdateB-'=[{ 1 1/2 | | -1/3-1/3 |= =1/2 1/2 
—3/2) \ 2/3 -1/31 aif 179 3/0 


11. Itis satisfied that /,_ < xg < ug. The optimal solution and value are 


x* = (3/2, —3/2, 0, 9/2,0, -3)7,  f* = 3/2 + 2(-3/2) = —3/2. 


8.4 Generalized Dual Simplex Algorithm with 
Bound-Flipping 


A so-called bound-flipping tactic can improve effectiveness of the generalized dual 
simplex algorithm significantly. In fact, it might be the main cause for the dual 
simplex algorithm to outperform its primal counterpart at present (Kirillova et al. 
1979; Fourer 1994; Kostina 2002; Maros 2003a,b; Koberstein 2008). 

Let (y, Z) be the current dual feasible basic solution and let x be the associated 
primal solution. Assume that a row index p has been determined by (8.21) and that 
a column index g determined by the minimum-ratio test (8.52). Let the nonbasic 
variable x, change from the current value xg (going up or down) toward the other 
bound, while keeping the other nonbasic variables unchanged. For the basic variable 
xj, to attain the violated bound, the value of x, could fall either within the range 
between the lower and upper bounds or beyond the other bound. In the latter case, 
it is favorable to execute the following so-called bound-flipping tactic: 

Fix the value of x, on the other bound and update values of basic variables 
accordingly; then find new column index q that attains the second minimum ratio, 
and do the same thing again, until the value of x;, attains the violated bound while 
the current value of x, falls within the range between its lower and upper bounds. 
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Then, a normal step is taken by dropping x;, from and entering x, to the basis, 
and the primal and dual solutions are updated. It is seen that the dual feasibility is 
maintained. 

The bound-flipping tactic is embedded in the following subalgorithm. 


Subalgorithm 8.4.1 (Bound-Flipping Subalgorithm) This algorithm provides the 
pivot column index gq, dual stepsize 8, and carries out related computations. 


Set j = 0, v = 0, and computer; = —zj/oj;, Vj € J. 
. Setv=Jjt+l. 
Set v = vu + ddgq. 
Set X, = me Ht 2 =, 
Ig, If xg = Ug. 


. Update: pp = pp — |doq]. 
Determine g and rg such that rg = minjes rj. 
. Compute Ax, by (8.57). 
. Update: J = J\{q}. 
. Go to step 13 if J = J. 
10. Compute 6 = ee alas Ix = las 
Iq — Ug, If xq = Ug. 
11. Go to step 2 if |Axg| = [64]. 
12. Set B=rg. 
13. Compute u = B7!v, and update: xg = Xp —u. 
14. Return. 


CONAN RF WN 


The following master algorithm is a slight modification of Algorithm 8.3.1. 


Algorithm 8.4.1 (Generalized Dual Simplex Algorithm: Bound-Flipping) Ini- 
tial: (B, N), B7!, 9, Z,x satisfying (8.28), (8.33), and (8.34). This algorithm solves 
the bounded-variable problem (8.3). 


. Select row index p by (8.21) together with (8.20). 

. Stop if pp, = 0 (optimality achieved). 

. Compute oy by (8.46) together with (8.45). 

. Stop if J defined by (8.51) is empty (Dual unbounded or primal infeasible). 
Determine column index g by (8.52). 

. Compute Axg by (8.57). 

. Set J = J\{q}. 

. If J =¥G, goto is 12. 

- - If xg =I, 


CIDWARWNE 


\o 


_ Compute 5 = 
eek i qr If iq = Ua- 


. If |Axg| = [6], call / Aue aie 8.4.1. 

. Compute dg = B~ lay. 

. Update x by (8.58). 

. Update y, zy, am by (8.49) together with (8.50). 
. Update B~! by (3.21). 

. Update (B, N) by exchanging j, and q. 


el el 
Ak WNrF © 
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16. Go to step 1. 


The bound-flipping adds extra work, involving a linear system (in step 13 of 
Algorithm 8.4.1). This is tolerable, compared with the return. Since the associated 
dual stepsize is usually much larger than that without bound-flipping, the increment 
in objective value is also much larger, and hence the number of iterations required 
usually reduces significantly. The bound-flipping tactic is essential in the current 


dual simplex codes. 


8 Generalized Simplex Method 


Example 8.4.1 Solve the following problem by Algorithm 8.4.1: 


min f = —x; + 2x3 + 3x4, 


s.t. —2xy +x. +43 +24 = -2, 
x1 —x3 +x4 +45 = 1, 
x] —2x3 —3x4 tx6 = 0, 
O<x, <2, -6<x< 10, <x3<7, 
l<x4<5, 2<x5<6, —-l<x<6. 


Answer Initial: B = 


1(2,5,.6) N= 11,34), 8 t= 7, ay SOO 05, 15), 


xz = (1,—-2, 1)", ¥ = 0,0,0), Zy = (-1,2,3)", f =1. 


Iteration 1: 


J ={I, 2}. 


. max{0, 2 — (—2), 0} = 4, p = 2, x5 leaves the basis. 
-h= —sign(2) B-Te2 


= (0,—-1,0)', oy = —Nth = (1, —1, 1)!. 


6B = min{—(—1)/1, —2/-1} =1, g=1. 
Ax) = —p2/\o\| = —4/1 = —4. 
Josie we6. 


ON ANWAWS 


6=l—uy =O0-2=-2. 


10. |Ax1| > [6], so call Algorithm 8.4.1. 


1) j=0,v=0, n= 


2) j=j+l=l; 


9/1 = 2: 


3) vu=v+6da, = (—2)(—2, 1, 1)? = (4, —2, —2)7; 


4) x} =1,=0; 


5) p2 = p2 — |6o)] = 4-2x 1 = 2, 


6) q=2,; 


7) Ax2 = —/2/|o2| = 


8) J = J\{2} =G; 
12) B=nm=2; 


—2/(-)) = 2; 


13) u= Boy = (4, -2, 2)"; 
Xp = Xp —u = (1,2, 1)" — (4, -2, -2)7 = (3, 0, 3)); 


14) Return. 


11. & = Boa) = (1, -1, —2)". 


12. xp = (—3, 0,3)? (—5 
in = (0,0, 1)" + 0, 2,0) = 0,2, 7. 


20,1, —2)' = 
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13. y = (0,0, 0)" + 2(0, —1, 0)? = (0, —2, 0)", 
Zn = Zn + Bon = (—1,2,3)' + 201,-1, 1)? = G,0,5)7, Z, = 
sign(p2)B = 2. 
11 
14. B= -1 
= a 
16. Xz = (—5,2,7)', nv = O-), 2-5, 15)", Zw = 1, 2,5), 
B = {2,3, 6}, N = {1,5, 4}. 


Iteration 2: 


1. max{0, 0, 7 — 6} = 1, p = 3, x¢ leaves the basis. 
3. h = —sign(p3)B~'e3 = (0, —2, 1)', oy = —N*h = (1, 2,5)". 
4. J = 9%, hence the problem is dual unbounded, or primal infeasible. 


If bound-flipping had not been used, there would be much more iterations required. 
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Chapter 9 ®) 
Decomposition Method od 


Solving large-scale LP problems is a challenging task, putting forward high 
requirements on the algorithms’ efficiency, storage, and numerical stability. The 
decomposition method divides a large-scale LP problem into relatively small ones 
to cope with normal LP solvers. 

This can be done because the objective function and nonnegativity constraints 
are variable separable, i.e., can be divided into several independent groups. If, in 
addition, the system of constraint equations is also variable-separable, then the 
large-scale problem can be handled by solving smaller problems. This chapter 
presents D-W decomposition and its illustration, its economic interpretation, Ben- 
ders decomposition and its illustration, and dual Benders decomposition. A new 
decomposition principle will be introduced in the final chapter of Part II. 

Let us bring up the following example: 


min f = (c!)?x!4.---+4 (c%)T xk, 


s.t. Dx! = h}, 
Dox? =h?, 
(9.1) 
Dyx* = hk, 
xi, xk > 0, 


where the coefficient matrix is of a block diagonal structure: orders of submatrices 
Dj; and dimensions of subvectors ci xi hi, j =1,...,k, are consistent, that is, 
each pair of c; and x; share the same dimension, equal to the number of columns of 
Dj, and the dimension of b; is equal to the number of rows of D;. 

The feasible region of the problem is 


P’ = {((x')",..., 5") | Djxt =hi, 27 > 0, jf =1,..., 4}, 
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or, otherwise, denoted in the form of the “Cartesian product”: 
P’=P!x...x PE, Pi = {xi | Djxi =hi, xi > 0}, FH,...,k. 


Since variables and constraints of the problem can be separated into independent k 
sets, the problem comes down to solving k smaller problems, i.e., 


min (c/)'x/, 


. : . j=1,...,k. 
st. Djxi =h!, x/>0, f 


If there is a feasible basic solution x/, associated objective value f/, to each of 
the small problems, there is a feasible basic solution, associated with corresponding 
objective value, to the original problem, i.e., 


z=(@')'”,...,.@597, fafit---+f (9.2) 


The same is valid with optimal basic solutions and objective values. On the other 
hand, if some of the small problems are unbounded, the original is unbounded too; 
if required, it is not difficult to determine the associated descent extreme direction 
(Proposition 3.3.2). 

Although such types of problems would not be encountered in practice very 
often, there are problems with some partially separable structures, which can be 
solved by methods, such as D-W decomposition and Benders decomposition, as 
introduced in this chapter. 


9.1 D-W Decomposition 


The Dantzig—Wolfe (D-W) decomposition (1960, 1961) partitions constraints into 
two sets and handles a master problem and a subprogram alternately. The master 
problem is expressed in terms of vertices and extreme directions of the feasible 
region, associated with one of the sets. Throughout the solution process, the 
subprogram takes the set as its constraints with its objective function varying 
iteration by iteration. It is solved to determine a column of the master problem to 
enter the basis or claim optimality of the original problem achieved. As a result, 
the LP problem is solved by dealing with two smaller programs. Further division is 
possible if the set of constraints of the subprogram has a separable structure. 

By partitioning the constraints into two sets, the standard LP problem can be 
written as 


min f =c'x, 


s.t. Ax =b, (9.3) 
Hx=h, x>0, 
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where A € R”™", HE R™*". hER™ he R™, 
Introduce polyhedral convex set 


P={xeR" | Hx=h, x = 0}. 


Let P be nonempty and possess the following sets of vertices and extreme 
directions, respectively: 


U={u!,...,u}, V={vl,...,u'}. 


According to the Representation Theorem 2.4.1, it holds that 


Ss t Ss 
P= es) on +) Bt Ye She 20; tH 10.48, FSD, FH hed, 
i=1 j=l i=1 


Substituting the expression of x into min{c! x | Ax = b}, we convert (9.3) into the 
standard problem with respect to variables a;, 6, i.e., 


Ss t 
Ming;g; f = So (ctu! ou + yew )R 


i=l j=l 


Ss t 
s.t. S (Au! or + $0 (Av!) Bj = b, 


i=1 j=l 


s 
i=1 


aj, Bj >0, i=1,...,5, fHl,...,t, 


(9.4) 


which is called master problem. 
The relation between feasible solutions to the master problem and to the original 
problem is determined by: 


x= So qu + > By, (9.5) 


corresponding to equal feasible values. Respectively, the feasible/optimal solutions 
of the master problem correspond to those of the original problem. In general, 
however, the latter is not a basic solution even if the former is. 

Let us focus on the master problem alone at the moment. The difficulty seems 
that the number s + ¢ of columns of the master problem is usually much larger than 
the number m-+1 of its rows, even if the number of rows would be much less than the 
number m + m1 of the rows of the original problem; moreover, vertices and extreme 
directions of P are used in the expression of the coefficient matrix and the objective 
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of the master problem. Fortunately, D-W decomposition can generate an entering 

column (with a leaving index determined), which allows simplex iteration to make 

basis change. For this reason, the method is often termed column generation. 
Without loss of generality, assume that the current basis matrix is 


Au! ..- Au’ Av! ..- Av’ 
B= 9.6 
( 1 --. 1 OO. O ) (6) 


the number of columns of which is s’ + t! = m + 1. The basic components of the 
associated feasible solution are defined by system 


Bay, ..., Qs, BL, woe, Bye — (7) P 


Denote the feasible solution by: 
a>0, i=l,...,5; Bj, jol,...,t. 


Then the associated feasible solution and objective value of the original problem are 


s! t 
E= Dla + VIB. f= Docu ae + Dev) Bj. (9.7) 
= i=] j=l 


The simplex multipliers (y, 7), corresponding to the first m rows and the bottom 
row, are defined by system 


B" 2) =n (9.8) 


where Cg consists of costs of the master problem, corresponding to basic columns. 
Check numbers (including the vanished reduced costs) are then 


clul — (Au')'5 - 7, i=1,...,5, 


clu! — (Av!) y, el eth: 


If the reduced costs are all nonnegative, then optimality of the original problem is 
achieved, and (9.7) gives its optimal solution and objective value; otherwise, some 
nonbasic column corresponding to a negative reduced cost is selected to enter the 
basis. 

To avoid computing all the reduced costs, D-W decomposition solves a subpro- 
gram with P as its feasible region instead, 1.e., 


min £ = (c — (A)"y)"x — y, 


st. Hx=h, x>0. ee) 
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Let us examine the relationship between the subprogram and the master as well 
as the original problems. If the feasible region P of the subprogram is empty, 
the master and the original programs are both infeasible. Otherwise, we have the 
following result. 


Lemma 9.1.1 Let f* be the optimal value of the original problem. If f is the 
current objective value of the master problem, and ¢* is the optimal value of the 
subprogram, then it holds that 


pre, as (9.10) 
Proof It is clear that f* < f, and hence it is only needed to show 
feoas. (9.11) 


Let x* be optimal solution of the original problem. Since it is also a feasible 
solution to the subprogram, we have 


(c— (A)"y)Tx* — p > e*. 


Note that B~!(h™, 1)" is a basic solution. From the preceding expression, Ax* = b 
and (y, y) satisfying system (9.8), it follows that 


2 = a 7 Tp,-1 [0 = 
ft sclx* > PAx* +740" = (PR+P) +E" = EBB (Tact Fer 


which gives (9.11). oO 


In each iteration, therefore, it is possible to estimate the optimal value of the 
original problem by giving lower and upper bounds by (9.10). It holds by (9.11) that 


fT a0". 


If f is taken as an approximately optimal value of the original problem, therefore, 
the associated absolute error bound is —¢*. If f 4 0, it follows further that 


(f —fO/Ifl <-o*V/IFl, 


the right-hand side of which gives a relative error bound. 


Theorem 9.1.2 [f the optimal value of the subprogram vanishes, optimality of the 
original problem is achieved. 


Proof By Lemma 9.1.1 and f — f* > 0, it is known that ¢* < 0, that is, the 
optimal value of the subprogram is less than or equal to 0. If it is equal to zero, 
then f < f* < f, which implies that optimality of the master and of the original 
problem is achieved. Oo 
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by 


If the feasible region P of the subprogram is nonempty, consequently, solving it 
the simplex method leads to one of the following two cases: 


(i) A vertex optimal solution, say u*, of the subprogram, is reached. 


(ii) 


If the optimal value vanishes, optimality of the original problem is achieved, 
and we are done; otherwise, a column 


* 
wi = i ) (9.12) 
is generated to enter the basis. 


It is detected that the subprogram is unbounded. Proposition 3.3.2 In this case, 
a descent extreme direction, say v* € V, of P can be determined such that 
(Proposition 3.3.2) 

(c — (A) 3) Tv* <0, 


implying that the associated reduced costs of the master problem are negative. 


Then, a column 
* 
w' = i ) (9.13) 


is generated to enter the basis. To avoid repetition, other simplex steps, such as 
the determination of a leaving column, and etc. will not be described here. 


The overall steps are summarized into the following algorithm. 


Algorithm 9.1.1 (Dantzig—Wolfe Decomposition) Initial: basis B of the master 
problem, basic components aj, Bj of the feasible basic solution, and associated 
objective value f. This algorithm solves LP problem (9.3). 


1. 
2. 
3. 


( 


Solve system (9.8) for (y, y). 
Call the simplex algorithm to solve subprogram (9.9). 
If an optimal solution u* of the subprogram is obtained, then: 


1) If the associated optimal value ¢* < 0, generate an entering basic column w’ 
by (9.12), and go to step 5. 


(2) Else compute the optimal solution and objective value of the original problem 


OoMmArANKNMN 


by (9.7), and stop. 


. If it is detected that the subprogram is unbounded, generate a new basic column 


w’ by (9.13). 


. Solve system Bw = w’ for w. 

. Stop if w < 0 (the original problem is unbounded). 

. Determine a stepsize a and leaving index by the minimum-ratio test. 
. Update B, a;, Bj. and f. 

. Go to step 1. 
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Note When the optimal value ¢* of the subprogram is close to 0, terminating the 
preceding algorithm gives an approximate optimal solution to the original problem 
(Lemma 9.1.1). 

The algorithm terminates if nondegeneracy is assumed. 

It is noticeable that the new reduced cost, corresponding to a leaving column, 
is certainly positive (see Proposition 3.1.1 and its proof). In each iteration of 
Algorithm 9.1.1, therefore, actually solved is a restricted master problem with m+ 2 
columns, consisting of basic columns and the entering column. The decomposition 
algorithm can be modified to allow a restricted master problem including more 
columns. Thus, we redefine the following. 


Definition 9.1.1 The problem consisting of k columns of the master problem is a 
restricted subprogram, where m + 2 < k < s+, provided these columns have full 
row rank m + 1. 


Then we have the following algorithm. 


Algorithm 9.1.2 (Modified Dantzig—Wolfe Decomposition) The same as Algo- 
rithm 9.1.1, except the restricted master problem is solved before it is updated (by 
generating a new entering and determining a leaving index). 


There are multiple ways to construct such an initial restricted master problem. 
For example, taking any m + | independent columns to form an initial basis, we 
add new entering columns successively until the number of columns attaining k 
or, otherwise, add the nonbasic columns, corresponding to the smallest k — m — 
1 reduced costs, after the number of columns is still less than k but close to the 
computer’s storage limit. 


9.1.1 Starting-Up of D-W Decomposition 


At first, we attempt to find a vertex, say ul, of P={x eR" | Hx =h, x > O}. 
If it does not exist, the original LP problem is clearly infeasible. Assume that u! 
is available. We introduce m artificial variables 07, | = 1,...,m to construct the 
following auxiliary Phase-I master problem: 


m 


MING; ,«;,B; w= ) Ol, 
i=l 
m 


Ss t 
s.t. Y > feo; + Y > (Au' oy + Y“(Av!) Bj = d, 


I=1 i=l j=l 


Ss 
w=, 
i=1 


Ol; a;, Bj = 9, l=1,...,.m,i=1,...,5, j=l,...,t, 


(9.14) 
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where for/ = 1,...,m, if bj — (Au'), > 0, +e is taken; else, —e; taken. Thus, a, 
together with o7, / = 1,..., m, constitute a set of basic variables. Basic components 
of the feasible basic solution are 


a&=1, o = |b)—(Au')|>0, 1=1,...,m. 


Then Algorithm 9.1.1 can get itself started from the basis consisting of columns 
associated with these basic components. 

It is clear that there exists an optimal solution, corresponding to a nonnegative 
optimal value, to the auxiliary problem: if the optimal value is greater than zero, the 
original problem is infeasible; if, otherwise, the optimal value vanishes, a feasible 
solution of the master problem comes from deleting all artificial components from 
the auxiliary optimal solution, giving a feasible basic solution to the master problem, 
which can be taken as an initial one for Phase-II, if artificial components are all 
nonbasic. In the case when some artificial variables are basic, additional iterations 
are needed to force them to leave the basis, just as following-up steps described in 
Sect. 3.2. 


9.2 Illustration of D-W Decomposition 


In principle, D-W decomposition can handle general standard LP problems, and the 
bisection of constraints can be arbitrary. In terms of effectiveness, nevertheless, its 
performance varies with problems of different structures. Many problems coming 
from practice are often of some (or partial) structures that are amenable to D-W 
decomposition. 

Models involving block structure may belong to this category, as illustrated by 
the example below. 


Example 9.2.1 Solve the following problem by the D-W decomposition: 


min f = —x2 + x3 — 2x4 — 3x5 + x6 — x7 + x8, 
s.t. xy — 3x2 — x3 +x4 + 2x5 + x6 — x7 — 3xg = 2, 
x1 + 4x3 +24 = 1, 
x2 +4x3-—x4 = 4, 
x5 + 3x7 —xg = 1, 
x6 + x7 — 3xg = 2, 
xj>0, fol,...,8 


Answer The first equality constraint is a coupling one, corresponding to a master 
problem of form (9.19), while the others correspond to a subproblem of the form 
(G21); 
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The second and third equality constraints are 


xy +4x34+ x4 = 1, 


x2+x3—-X%4=4, x7 =0, fH l,...,4, 


which has a basic solution xj = 1, x2 = 4, x3 = x4 = O, and the fourth and fifth 
are 


x5 + 3x7 —xg = 1, 


Xg6+x7—-3xg=2, xj >0, j=5,...,8, 


which has a basic solution x5 = 1,x6 = 2, x7 = xg = O. Therefore, there is a 
feasible basic solution u! = (1, 4, 0, 0, 1, 2, 0, 0)? to the subprogram. 
Accordingly, 


bj —(Au'); = 2—(1, —3, -1, 1, 2, 1, -1, —3)(1, 4, 0, 0, 1,2,0,0)' =24+7=9>0. 


Thereby, we construct an auxiliary master problem of form (9.14), ie., 


min oj, 


5 t 
s.t. 01 — Tay + ) (Au! ay + (Av!) Bj = 2, 


: ia = (9.15) 
So ai = 1, 

i=l 
o1,a;,B;>0, i=1,...,5, jHl,...,t, 


where A= (1,=3, 115 9, 1,—1,=3). 


The basis matrix is B = (; 1) . Solving system B(o}, a1)! = (2, 1)" leads 


; : ; : O 9 : 
to the feasible basic solution, whose basic components are a ) = ( ) , With 
a 


auxiliary objective value w = 1. 


Phase-I: Call D-W Decomposition 9.1.1 to solve the auxiliary master problem 
(9.15). 
Iterations 1: 


1. Solving B'(y, y)' = (1, 0)! gives simplex multipliers (¥, 7) = (1, 7). 
2. The objection function of the subprogram is 
$=(c-(A)"y)'x-7 
=(0, 21, 1,2, 53211, D0, =3, 1,12, le =4)e7 
= (—1, 2,2, —3, —5, 0,0, 4)x — 7. 
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To be solved separately, the subprogram is decomposed into the following two 
smaller programs: 


(i) 
min —x; + 2x2 + 2x3 — 3x4, 
st. xy +443 4+ x4 = 1, 
x2+%x3-x4=4, x7 >0, j= l,...,4. 

Taking basis and nonbasis B = {1,2}, N = {3, 4} (B~! = 1) determines 
the feasible basic solution xg = (1,4)' > 0, Xv = (0, 0)!, and the reduced 
costs 

T 
7 2 4 1 -1 4 
—cy —N'™B7'cp = = . 
annen—wTIee=(_3)-(7 4) (2) =(0) 

Since min{4,0} = 0 > O, the following optimal vertex solution with 
objective value is obtained: 

*=(1,4,0,0)', 0 =7. 
(ii) 


min —5xs5 + +4xs, 
s.t. x5 + 3x7-—xg = 1, 
Xo +xX7—3xg=2, xj =0, j=5,...,8. 


Taking basis and nonbasis B = {5,6}, N = {7,8} (B-! = 1) determines 


feasible basic solution xg = (1, 2)t >0, xy = (0, o)T, and the reduced costs 


ty = ey — N™B-leg = ({) = (; =) (%) = ("). 


Thus min{15, —1} = —1 <0, q = 8. Since B~!ag = (—1, —3)' < 0, 
the subprogram is unbounded. 

4. The feasible region of the subprogram has descent extreme direction v! = 

(0, 0,0, 0, 1, 3,0, 1). 

Generate entering column w’ = (2, 0)t by (9.13). 

. w= (2,0)! £0. 
.a=min{9/2}, p=1. 
8. (G1, @)! = (9, 1)T — (9/2)(2, 0)? = (0, 1)T. 


san 


Be & =) (A, a)" = 9/2, DT, 
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f = 0,-1, 1, —2, —3, 1, —1, 1)(0, 0, 0, 0, 1, 3, 0, 1)7(9/2) 

+(0, —1, 1, —2, —3, 1, -1, 1), 4, 0, 0, 1, 2, 0, 0)? = —1/2. 

The only artificial variable o; has left the basis, hence Phase-I is finished. 
Phase-II: Call D-W Decomposition 9.1.1 to solve the master problem. 
Iterations 2: 

1. ég = (clv', clu!)? = 1, —5)". 

Solve B'(y,y)' = (1,—5)! for simplex multipliers (7,7) = 
(1/2, —3/2). 
2. The objective function of the subprogram is 


¢=(c—-(A)y)x -7 
= ((0, —1, 1, —2, —3, 1, —1, 1) — d, —3, —1, 1, 2, 1, —1, —3)(1/2))x + 3/2 
= (—1/2, 1/2, 3/2, —5/2, —4, 1/2, —1/2, 5/2)x + 3/2. 


(i) 


min —1/2x; + 1/2x2 + 3/2x3 — 5/2x4, 
s.t. xy +443 4+ x4= 1, 
x2+%x3-x4=4, x7 >0, fj=l,...,4, 


which has the optimal vertex solution (x1, 2, %3,x4)' = (0,5,0, 1)" 
with optimal value ¢; = 0. 


(ii) 


min —4x5 + 1/2x6 — 1/2x7 + 5/2xs, 
s.t. x5 + 3x7 -—xg = 1, 
Xo +x7—3xg=2, xj =0, j=5,...,8 


has optimal vertex solution (x5, x6, X7, xg)? = (1, 2,0, 0)? with optimal 
value ¢; = —3. 

Thereby, the subprogram has the following optimal solution and 
optimal value: 


u> = (0,5,0,1,1,2,0,0)', ¢c*=0-—3+43/2 =—-3/2 <0. 


3(1). ¢* < 0. Generate entering column w’ = (—10, bt by (9.12). 
5. w = (—-3/2,1)7 £0. 
7. a =min{l/1}, p = 2. 


8. (Bi, a1)" = (9/2, 1)? — (—3/2, 1)? = 6,0)". B= [ a 


(B1, 2)" = (6, 17. f = f +ae* = —1/2—3/2 = -2. 
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Iterations 3: 
1. ég = (clv!, clu?)? = (1, -8)". 
Solve B'(y, vy)’ = (1, —8)! for simplex multipliers (¥, 7) = (1/2, —3). 
2. The objective of the subprogram is 
b= (c—(A)'y)'x —7 
= ((0, —1, 1, —2, —3, 1, —1, 1) — (1, —3, —1, 1, 2, 1, -1, —3)(1/2))x +3 
= (—1/2, 1/2, 3/2, —5/2, —4, 1/2, —1/2, 5/2)x +3. 


(i) The problem is the same as that in the previous iteration, with optimal 
vertex solution and objective value 


(%1, X2, X3,X4)' = (0,5,0,1)', % =0. 


(ii) The problem is the same as that in the previous iteration, with optimal 
vertex solution 
(X5, ¥6, X7,X8)! = (1, 2, 0, 0)" and optimal value ¢; = —3. 
The optimal vertex solution and optimal value to the subprogram are 


u>(0,5,0,1,1,2,0,0)', ¢c*=0—-343=0. 


3(2) Since ¢* = 0, an optimal solution and optimal value to the original problem 
are obtained, i.e., 


x* = (0, 0, 0, 0, 1, 3, 0, 1)'(6) + (0, 5, 0, 1, 1, 2, 0, 0)7 
= (0, 5, 0, 1, 7, 20,0,0)', f*= f= -2. 


Another suitable type of problems have staircase structure close to block, like 
the following instance, which can be solved by the D-W decomposition in a nested 
manner: 


min f = (cl)Tx! + (2)Tx?2 + (3) x3 + (ce) Tx4, 
S.t. Dix! = h!, 

Agx! + Doyx? = b?, 

A32x? + D33x3 = D?, 

Aa3x? + Dagx* = b+, 

xix? x3, x4 > 0, 


(9.16) 


where x/, j = 1,2,3,4 are vectors of appropriate dimensions consistent with the 
preceding matrices. 
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Partition the equality constraints of the above problem into two parts, denoted by 
{t = 1} and {t = 2, 3, 4}, respectively. Let {u'} and {v/} be, respectively, the vertex 
set and extreme direction set of the set of solutions to the latter part {t = 2, 3, 4}, 
Le., 


Aix! + Dyx* = b?, 
A32x7 + D33x? = b3, 
Aa3x? + Daax* = b4, 
x) x? x3 x4 > 0. 
According to the Representation Theorem 2.4.1, the solution to the preceding 
can be expressed as 


S t S 
x=) aju' +) > Bj, yeah a,j; >0,i=1,...,5, j=Hl,...,¢. 
j=t j=l i=l 


(9.17) 
Introduce partition 
re ui wat 
x . i ui? : . j vi? 
x= ele u= We ii Perera ae wife Pea ery | 
x4 yi4 yi4 


Substituting (9.17) to the first part {t = 1} leads to D-W master problem 


ming;.f; ¥ = yey ze (c?) Tui? a8 (c3)Tu!3 fe (ct) ul); 
t=1 
t 
+ ocelytull + (7)Tv? + (Tv? + Tv!) Bj, 


j=l 


s t 
s.t. Yo Diuru'la; + S5 Div!" B; =", 


i=1 j=l 


s 
doa = 1, 
i=1 


aj,8; =>0, i=l,...,s, jHl,...,t. 
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Assume that the current restricted master problem has been solved, and (y1, ¥1) is 
the optimal simplex multiplier. The constructed subprogram 


min (c! — (Dy) 91) Tx! + (c2)Tx? + (c3)Tx3 — 1, 
S.t. Az x! + Dox? 1 b*, 

A32x* + D33x7 = b, 

Aa3x? + Dagx* = b*, 

xt, x2, x3, x4 > 0. 


just likes the original problem, but the number of stairs reduces by 1. Then apply 
D-W decomposition again by partitioning it into two parts, {t = 2} and {t = 3, 4}, 
and do as previously until the stairs are reduced to 1, and hence the subprogram is 
easier to solve. 


9.3. Economic Interpretation of D-W Decomposition 


The D-W decomposition can be interpreted differently, depending on various 
application backgrounds. Using this method to realize the optimal allocation of 
limited resources is a typical example. 

Assume that a company has k plants. It is required to determine an output 
vector x/ for each plant j, whose production activities depend on restrictions upon 
available limited resources, such as manpower, facilities, materials, storage, etc., but 
not on inner restrictions of the other plants. In mathematical terms, x/ should satisfy 
constraints 


Djxi =hi, x/ >0, PSH) ees 


where Dj € R™*"i (mj < nj), termed consumption matrix, reflects the 
consumption for a unit product, and h/ is vector of available resource. This part of 
constraints is of block structure with variables and constraints separable. In addition, 
there are common resources to be shared by the k plants, which can be expressed by 
a set of so-called coupling constraints: 


Aix! +... + Apx* = b, 


where Aj ER™*"), f=l,...,k. 
To minimize the total cost of production, we solve the following LP model: 


min f = (Copa) ae ae See ae Con ee ae 
st. Ayxi+---+Azpxk = b, (9.18) 
Djxi=hi, x4 >0, joHl,...,k. 


9.3 Economic Interpretation of D-W Decomposition 239 


It is advantageous to use D-W decomposition if the preceding model is of large 
scale, especially when there are a large number of plants associated with large 
consumption matrices: the company can focus on a smaller problem to realize an 
optimal allocation of resources by ignoring the details of the inner restrictions on 
the plants. 

For simplicity, one might assume that the feasible regions 


PIA {xi | Djxi shi, xi 0}, faly...sk, 


corresponding to the plants, are all bounded (actually, available resources are always 
limited). Introduce notation 


D; h! 
A=(A,,...,Ag), H= ; 
Dx hk 
Thus, it is clear that 
Pa tceR" | Hx=h, x>0}=P!x.-- x Pk 


is bounded. Denote the vertex set of P by U = {u!, ..., u°}. Based on Representa- 
tion Theorem 2.4.1, we obtain the following master problem: 


Ss 
ming, f = )o(clu')ai, 
i=1 


s.t. So (Au')a; = b, eae 


The number of rows of the preceding is less than the number of rows of the original 
problem, but the number of its columns would be quite large, compared with the 
original problem, letting alone unrealistic computation of the whole vertex set U. 
Nevertheless, the company can deal with a restricted master problem instead, 
involving only some k (m +2 <k <5) columns (with rank m + 1) of the master 
problem, ignoring most of the inner restrictions of the plants. Now denote by U the 
vertex set, resulting from U by deleting vertices corresponding to the k columns. 
Assume that optimality of the restricted master problem was achieved and that 
(y, Y) is the optimal simplex multipliers, associated with the first m rows and the 
bottom row of the constraint system. In economic terms, (y, 7) is the shadow price 
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of resources (Sect. 5.6), from which the reduced costs corresponding to set U follow, 
1.€., 


cul — (Au’) 3 —p =(c-(A) pu 7: ul eS. (9.20) 


A negative reduced cost implies that increasing the associated nonbasic variable 
from O results in decrement of the total cost of production. Typically, the most 
negative reduced cost is selected to enter the basis. The difficulty is that computing 
the reduced costs directly by (9.20) requires all vertices of P available, while the 
company does not know the inner details of the plants. 

The trick to get rid of this difficulty is to solve the following subprogram by the 
simplex algorithm, i.e., 


min ¢ = (c — (A)"y)"x — 7, (9.21) 
st. x EP. 
where the objective is related to the shadow price (y, y). 

Since the P is bounded, solving (9.21) renders an optimal vertex solution, say 
u*, associated with the optimal value, say ¢*, equal to the minimum reduced cost of 
the restricted master problem. If ¢* > 0, optimality of the master problem as well as 
the original problem is achieved, so is an optimal allocation scheme of resources. If, 
otherwise, ¢* < 0, then u* corresponds to the most negative reduced cost, hence the 
restricted master problem can be updated by entering to basis the master problem’s 
column corresponding to u*, and dropping from basis the column determined by the 
minimum-ratio test. Then an iteration is complete. 

It is noted that subprogram (9.21) can be further decomposed into k smaller 
programs as 


min (c/ — Aly)x/ —/, 
ets co 
s.t. x/ € Pl, 
Once optimal basic solutions to these programs are obtained, an optimal vertex 
solution of form (9.2) to the subprogram is readily available. 


9.4 Benders Decomposition 


The D-W decomposition presented previously converts an LP problem into two 
smaller ones by partitioning its rows into two parts. In contrast, Benders (1962) 
converts a problem into two smaller ones by partitioning its columns (variables) 
into two parts, one of which is an LP problem, and the other can belong to another 
category. It can be used to solve large-scale mixed LP programs, especially mixed- 
integer LP programs. Subsequently, Benders decomposition is widely applied to 
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dealing with stochastic programming programs and is further generalized to handle 
nonlinear programming programs (Geoffrion 1972). 
Consider 


maxz,y f(w) + b'y, 
s.t. F(m)+ Aly <e, (9.22) 
axel, 


where A € R”*", c € R", b € R"”. Scalar function f(s) and vector-valued 
function F(z) € R” and their domain IT Cc R” will be clarified later. 

For any fixed z, (9.22) is a LP problem with respect to variable y. Therefore, 
Benders decomposition firstly deems z as a parameter to realize a partition of 
variables. 


Lemma 9.4.1 Problem (9.22) is equivalent to the following problem: 


max, f (7) +maxy {b'y| Aly <c— F(m)}, (9.23) 
st. welins, 
where 
S={x | Aly+ F(x) < ch. 

Proof Itis clear that (9.22) is infeasible and unbounded and has an optimal solution 
if and only if (9.23) is infeasible and unbounded and has an optimal solution 
accordingly. 

For any fixed zr, the objective value of problem (9.23) is equal to the sum of f(z) 


and the optimal value of the following subprogram: 


D(a) : maxy bly, 
S.t. Aly <c— F(z). 


Let (z, y), a, and ¥ be an optimal solution to (9.22), (9.23), and D(z), 
respectively. Then it is clear that (j, 77) is a feasible solution to (9.22), and hence to 


f(@) +015 < f@ +b. (9.24) 
On the other hand, it is known that 
fG@) +05 > f@ + max {bTy | ATy <c— F@)}, 
y 
and y is clearly an optimal solution to D(z). Therefore, it holds that 


fH) + max {bTy| ATy <c— F(@)} = f(@) + b*9. 
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Combining the previous two formulas leads to 
FG) + DTS > ft) + bT5, 
which along with (9.24) gives 
fr) + BTS = fH) + bTy. 
Therefore programs (9.22) and (9.23) share the same optimal solutions and 


optimal value. oO 


Therefore, problem (9.22) is decomposed into the two smaller programs D(zr) 
and (9.23), respectively, involving p and q variables. Nevertheless, they are defined 
implicitly. To go further, consider the dual problem of D(z), i.e., 


P(x): min, ¢ = (c — F(s))'x, 
st. Ax=b, x>0. 


Proposition 9.4.1 [f the feasible region 
X={xeR" | Ax=b, x >0} 


of P(x) is empty, then (9.22) is infeasible or unbounded. 


Proof X = % means that P(z) is infeasible for any 7 € R”. According to the 
duality theory, problem D(zr) is infeasible or unbounded, hence (9.22) is infeasible 
or unbounded. oO 


In case when X = J, one can solve 


min, (c — F(s))'x, 
st. Ax =0, x>O0, 


by the simplex algorithm to determine whether (9.22) is infeasible or unbounded, if 
needed. If the preceding is unbounded, it can be asserted that (9.22) is infeasible, and 
if optimality is achieved, (9.22) is unbounded (see discussions made in Sect. 5.2). 

From now on, it is assumed that X 4 @, whose vertex and extreme direction sets 
are, respectively, denoted by 


Using preceding notation, we have the following result. 


Lemma 9.4.2. Given x, there is an optimal solution to problem P(x.) if and only if 


(c— F(r))'vi > 0, vi eV. (9.25) 
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Proof It is seen that (9.25) implies nonexistence of a descent unbounded direction 
in set X. Therefore, there exists an optimal solution to P(r). According to 
Theorem 2.5.5, there is an optimal vertex solution to it. 

Conversely, that there is an optimal solution to P(z) implies that (9.23) holds, 
because if there is some v* € V such that 


(c — F(r))'v* <0, 


then v“ is a descent extreme direction, hence P (2) is unbounded below, which 
contradicts that there exists an optimal solution. Oo 


Define the dual Benders master problem as follows (it will be clear why it is 
prefixed with “dual’): 


maxz,g f(7) + 8, 

s.t. g<(c—F(x))'u', ui €U, 
(c— F(x))'vi > 0, vie, 
nweETINS. 


(9.26) 


Theorem 9.4.1 (z, y) is an optimal solution to (9.22) if and only if y is an optimal 
solution to D(z) and (2, g) is an optimal solution to the dual Benders master 
problem. 


Proof It is clear that there is an optimal solution to (9.22) if and only if there is an 
optimal solution to the dual Benders master problem. 

According to Lemma 9.4.1, problem (9.22) is equivalent to (9.23). On the other 
hand, that there is optimal solution to problem (9.22) means that so is to D(z). By 
the strong dual theorem, there is optimal solution to P(z) with the same optimal 
value, 1.e., 


max {b'x | Aly <c— F(x)} =min {(c — F(a))'x | Ax =b, x > O}. 
y x 
Substitute the preceding to (9.23) leads to 


max, f(a) +min, {(c — F())'x | Ax =b, x > 0}, (9.27) 

st. wellns. 
By Theorem 2.5.5, if there is optimal solution to P(zr), then there is a vertex optimal 
solution to it. Therefore it holds that 


u! 


min {(c — F())'x | Ax =b, x > 0} = min(c — F(z))'u’. (9.28) 
x EU 
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Substituting the preceding to (9.27) and applying Lemma 9.4.2 lead to 


max, f(z) +min,icy(c — F(a))'u!, 
st. (c—F(x))'vi >0, wev, 
nxnelins, 


which gives (9.26). These equivalent relations ensure validity of Theorem 9.4.1. O 


According to the preceding theorem, what we should do next is to solve 
(9.26). However, it is not realistic to handle (9.26) directly, because it requires the 
availability of all vertices and extreme directions of X, let alone constraint m € S 
is given in implicit impression. To overcome this difficulty, a possible approach is 
to handle a relaxed problem, yielding from ignoring some of the constraints. Its 
optimal solution is also an optimal solution to the master problem if it is a feasible 
solution to the latter; otherwise, the relaxed problem is updated by adding some 
constraints. 

Assume that at some iteration, subsets of the vertex and extreme direction sets of 
X are known as 


eu, Vey, 
Define the dual Benders restricted master problem! by: 


mMaxz,g f(7) +8, 

st g<(c—F(x))'v', we’, 
(c— F(m))'vi > 0, vie V’, 
mel. 


(9.29) 


Theorem 9.4.4 Let (7, g) be an optimal solution to the dual Benders restricted 
master problem. 

Tf u* and ¢* are an optimal vertex solution and optimal value to the subprogram 
P(z), then: 


(i) If g > ¢*, then u* is a new vertex generated. 
(ii) If g = &*, then (Z, g) is an optimal solution to the dual Benders master 
problem. 


If P(at) is unbounded, then a new extreme direction is given. 


Proof Since u* and ¢* are optimal vertex solution and optimal value to the 
subprogram P (zc), it holds that 


c* = (c— F(a))'u* = min{(c — F(#))'u! | ul € US. (9.30) 


' The term by Dantzig—Wolfe is employed here. 
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Besides, z satisfies 
(c— F(z))'vi > 0, vi eV, (9.31) 


because otherwise there is an ascent extreme direction in X, hence P(z) is 
unbounded, which contradicts the existence of optimal solution. Moreover, it also 
holds that 


zens. (9.32) 


In fact, it is clear that zt € I]; andifz ¢ S, then it is derived that D(z) is infeasible, 
hence P(zr) is either infeasible or unbounded, which contradicts the existence of an 
optimal solution to P(z). 

Since (7, g) is an optimal solution to (9.29), it is obtained from (9.30) that 


f@+e= f(t) +min{(c— FG@))Tu! | wie UY 
> f(t) +min{(c— F(t))Tu! | ul € U} 
= fm) +6", 


hence g > ¢*. 
Consequently, only the following two cases arise: 


(i) g>*. 
In this case, from 
pe =C- Fa) 
and 
g=min{(c — F(z))'w! | ub eV} 
it follows that 
(c — F(z))'u* < min{(c — F(z))'u! | ui € U’S, 
implying that u* ¢ U’ is a new vertex. 
(ii) g=¢*. 
In view of (9.30), it can be obtained in this case that 
g=o*<(—F@)), weu, 
which together with (9.31) and (9.32) implies that (7, g) is a feasible solution 


to the dual Benders master problem. It is actually optimal because it is optimal 
to the relaxed problem of the dual Benders master problem. 
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If subprogram P(z) is unbounded, then an extreme direction v* is determined 
such that (Proposition 3.3.2) 


(c — F(z))'v* <0. 
Thus, from (7, g) satisfying constraints of the master problem, i.e., 
(c— F(t)’ > 0, wev’, 


it is known that v* ¢ V’ is a new extreme direction. oO 


Proposition 9.4.2 If the Benders restricted master problem is infeasible, then 
problem (9.22) is infeasible. 


Proof If M1 = @, (9.22) is clearly infeasible. If 1 + @, then infeasibility of the 
Benders restricted master problem implies that for any 2 € I, there is some v* € 
Vv’ c V such that (c — F(s))!v* < 0, hence for any z € II, problem P(z) is 
unbounded and D(zr) is infeasible, and the original problem is therefore infeasible. 


Oo 
The dual Benders master problem (9.26) can be converted to 
maxz,¢ f(t) + 8, 
s.t. Ww)’ Fa@)+e<(i)lce, uli eU,. 
@w)' Faye)", vl eV, * 
mzmelns. 
Accordingly, the dual Benders restricted master problem (9.29) becomes 
Maxz,¢ f(7) + 8, 
t i)T < (yi)T i f 
s.t Ww) F@)+g<t'jc, uiev’, (9.33) 


(v/)' Fir) < (v')'c, vi eV’, 
f(m)+e<M, (*) xell, 


where the constraint, marked by «, is added to rule out possibility of unboundedness, 
and M is a parameter large enough, so that such addition does not affect vertices of 
the original feasible region. We call the added constraint standby, because it is active 
only when the problem is unbounded. The parameter M can be dropped technically 
in real computations (see the example below). 

Based on Theorem 9.4.4 and Proposition 9.4.2, we proceed similarly as in 
Sect. 9.2 as follows: 

In each iteration, a Benders restricted master problem of form (9.33) is handled. 
If it is infeasible, the original problem is infeasible; if an optimal solution (7, g) 
is obtained, then an associated subprogram P(zr) is solved. When the subprogram 
is unbounded, a new extreme direction is generated (Proposition 3.3.2); when an 
optimal vertex u* and optimal value ¢* are reached, it can be determined whether 
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(7, g) is an optimal solution to the Benders master problem; if it is not, a new 
vertex u* is generated; if it is, then (7, y) is an optimal solution to the original 
problem (9.22), where y is optimal to D(z). In fact, there is no need for solving 
D(zr) actually, because y, as a dual solution associated with u*, can be obtained at 
the same time. 

In the case when optimality of (9.22) is not achieved, the restricted master 
problem is updated by adding a constraint, corresponding to the new vertex or 
extreme direction, and the next iteration is carried out. In such an iteration, the 
restricted master problem has one more constraint (this is why the method is often 
called row generation. Note that subprogram P(zr) is a standard LP problem, only 
its objective function changes in the solution process (with the constraints remaining 
unchanged). 

The overall steps are summarized in the following algorithm. 


Algorithm 9.4.1 (Benders Decomposition) Initial: dual Benders restricted master 
problem in form (9.33). This algorithm solves problem (9.22). 


1. Solve dual Benders restricted master problem. 

2. Stop if it is infeasible. 

3. If its optimal solution (7, g) is obtained, solve subprogram P (zr) by the simplex 
algorithm. 

4. If an optimal vertex solution u* and optimal value ¢* to P(z) are obtained, 
then: 


(1) Update (9.33) by adding the constraint, corresponding to the new vertex u*, 
and go to step lif g > ¢*. 
(2) Stop if g = ¢*. 


5. If subprogram P(z) is unbounded, update (9.33) by adding the constraint, 
associated with the generated new extreme direction. 
6. Go to step 1. 


Theorem 9.4.5 Assume that the dual Benders restricted master problem either has 
an optimal solution or is infeasible. If it and subprogram P (zc) are solved in finitely 
many iterations, Algorithm 9.4.1 terminates, at either: 


(i) Step 2, detecting that the original problem is infeasible or 
(ii) Step 4(2), achieving an optimal solution (x, y) to the original problem, where 
y and u* are complementary 


Proof The meaning of exit step 2 is directly from Proposition 9.4.2. According to 
Theorem 9.4.4, termination at step 4(2) gives optimal solution (7, g) to the dual 
Benders master problem; further, it is known by Theorem 9.4.1 that (7, y) is an 
optimal solution to the original problem, where y is the dual solution, associated 
with u*. Therefore, it is only needed to show termination of Algorithm 9.4.1. 

By Theorem 9.4.4, the restricted master problem has one more added constraint, 
associated with a vertex or extreme direction in each iteration. Since vertices and 
extreme directions in X are finite, the restricted master problem must become full 
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master problem in finitely many iterations, if the solution process does not terminate 
earlier. Oo 


Although the number of constraints of the dual Benders restricted master problem 
increases steadily, fortunately, it hardly becomes full dual Benders master problem. 
Usually, the process terminates when it is very far from the latter. 


9.5 Illustration of Benders Decomposition 


Efficiency in solution of the dual Benders restricted master problem is crucial to 
Algorithm 9.4.1, since it depends on characteristic of f(z) as well as the algorithm 
used to solve it. At present, there are quite good algorithms for the following cases, 
concerning (9.22): 


(i) f(t) = h'n, F(x) = H'x, h € R”", H € R"™", and ll Cc R” isa 
(bounded) polyhedron. 

In this case, the original problem (9.22) is a linear one. If z is a free variable 
(Il = R”), in fact, (9.22) is clearly a standard dual problem. As the dual 
restricted master problem and subprogram are linear, the simplex algorithm is 
a suitable tool for this purpose. 

(ii) The same as (i), except for adding some integer requirement to components of 
I. 

In this case, the original problem (9.22) is a mixed-integer LP problem, 
and hence the dual restricted master problem is a pure integer LP problem, 
except involving a free variable g. Since one inequality constraint is added 
to the restricted master problem in each iteration, the cutting plane method is 
amenable to be used (Sect. 11.2). 

(iti) f(a), F(z) are continuously differentiable functions on a bounded and closed 
convex set 1 c R”. 


In this case, the dual restricted master problem is a smooth convex problem, for 
which efficient algorithms are available (see, e.g., Sun and Yuan 2006). 

Cases (ii) and (iii) are beyond the scope of this book. The following discussions 
are focused on (1), related to LP. 

As for how to construct an initial dual restricted master problem, a direct way is 
to start from the feasible region 


X={xeR” | Ax=b, x =>0} 
of P(z). If X = JY, the original problem is infeasible or unbounded (Proposi- 
tion 9.4.1); otherwise, a feasible vertex, say “ey , can be determined. Therefore, 


allowing 


U'={u'}, V' =G, 
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leads to an initial dual restricted master problem in form (9.33). 


Benders decomposition is suitable for some programs, involving block diagonals, 
such as 


min h'x 4 (pyty! dese eee (BF) i 


S.t. Aln + Diy! < cq, 
Aln + Diy? <c 


A 
is) 
= 


Alx + +Df yk < 


where orders of submatrices D; and dimensions of subvectors b’, y',c', i = 
1,...,, are consistent, i.e., b; and y; are of dimension equal to the number of 
rows of D;, and c; of dimension equal to the number of columns of D;; z is of 
dimension equal to the number of rows of Aj. 

If one partitions the coefficient matrix as 


Ay Dt 
TT T 
H™ _ Ay AT _ D; 
T di 
Ay dD, 


then subprogram P(z‘) can be decomposed into k smaller programs, corresponding 
to Dx. 
Let us bring up the following example. 


Example 9.5.1 Solve the following problem by Benders Decomposition 9.4.1: 


max 27 + yj +4y2 + y3 + 2ya, 
st. wt+ty, <0, 
—37+)y2<-1, 
—m7+4yj+y2 <1, 
w+yi—yo25—2, 
2a + y3 < —3, 
zrt+tya<l, 
—7+3y3 +y4 <—-l, 
—3n — y3—3y4 <1. 


Answer Denote the problem by: 


maxz, y Al + by, 
S.t. H'n + Aly <c, 
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where 


h' = (2),b = (1,4, 1, 2)',¢ = (0, -1, 1, —2, —3, 1, -1, !. 


104 1000 0O 
011-1000 0O 
000 0103-1 
000 0011-3 


A=(1,-3,-1,1,2,1,-1,-3), A= 


There is a vertex in feasible region X = {x | Hx =b, x > 0} of Subprogram 
P(t), ie., 


u! = (1,4, 0,0, 1, 2,0, 0)?. 


(u')'H'x = (1,4, 0,0, 1, 2,0, 0)(1, —3, -1, 1, 2, 1, -1, -3)'x = —7z. 


(u')"e = (1,4, 0,0, 1, 2, 0,0)(0, —1, 1, -2, 3, 1, -1, 1)? = —S. 
The associated initial restricted master problem is 


maxz,, 27 + g, 
s.t. —TImn+g¢<—-5, 
2r7+g<M, (*) 


where (*) marks the standby constraint. 


Iteration 1: 
1. Solve the Benders restricted master problem by the simplex algorithm. 


Taking basis B = 7 4 ,B = te 7) gives the dual feasible 


11 1/9 7/9 
solution 
eeae -( Ae eee ean 
a) mM)” \-10/9+7/9M }° 


The associated primal solution is feasible, i.e., Bo} (2, 1)? = (0, 1)! > 0. Therefore 
optimality of the restricted master problem is achieved. 
3. The objective of Subprogram P (zr) is 


(@=Hay Sad, 3 eo) GO oh 3, 1-1. 18} 
= (—5/9 — 1/9M, 2/3 + 1/3M, 14/9 + 1/9M, —23/9 — 1/9M, 
—37/9 — 2/9M, 4/9 — 1/9M, —4/9 + 1/9M, 8/3 +1/3M). 
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As parameter M is large enough, the terms without M in the preceding can be 
omitted. For simplicity, the common factor M can be omitted as well; in other 
words, (9.34) can be replaced by 


Ua) 
g 1 

Subprogram P(z) is decomposed into the following two problems, as solved 
separately: 


(i) 


min —1/9x; + 1/3x2 + 1/9x3 — 1/9x4, 
s.t. x7 +443 + x4 = 1, 
x2+x3-x4=4, xj, >0, j=1,...,4. 


Using basis and nonbasis By = {1,2}, Ny = {3,4}. By! = I gives feasible 
basic solution xg, = (1, 4)" > 0, ¥y, = (0, 0)!. 


- a -1/9 
y= B, cn = ( 1/3)" 


T 

ae To 1/9 4 1 —-1/9\ _ (2/9 

a (is) (; i) ( 3) ~ (iis) 

min{2/9, 1/3} > 0. The reduced costs are all nonnegative, optimality is hence 
achieved. 

The optimal vertex solution is (x1, X2, x3, x4)? = (1,4,0, o)', with optimal 
value ¢; = 11/9. 
(ii) 


min —2/9x5 — 1/9x¢6 + 1/9x7 + 1/3xs, 
s.t. x5 + 3x7 —xg = 1, 
x6 +x7—-3xg=2, xj >0, fj =5,...,8. 


Take the basis and nonbasis B, = {5,6}, N2 = {7, 8}. Be =; 
The feasible basic solution is 
te, = 1,2)" 20, zm = G0)", 


: 5 =2/9 
yo = By ca, = ere 


T 
7 7 1/9 3-1 —2/9 8/9 
j= ir NI = _ = i 
ee ae «s & = i) (_ ) 
The minimum reduced cost is min{8/9, —2/9} = —2/9, q = 8. 
Since 4g = B; '(—1, —3)’ = (—1, —3)" < 0, P(t) is unbounded. The descent 


extreme direction is v* = (0, 0, 0, 0, 1, 3,0, 1)!. 
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5. (v*)'H'x = (0,0, 0,0, 1,3, 0, 1), —3, 1, 1, 2, 1, —1, —3)'x = 27. 


(v*)Tc = (0,0, 0, 0, 1, 3,0, 1), -1, 1, -2, -3, 1, -1, D7 = 1. 
The restricted master problem is then updated to 


Maxz,, 20 + g, 
s.t. —TIn+g¢<-5, 
Qn <1, 
2rn+g<M. (x) 
Iteration 2: 


1. Solve the restricted master problem. The basis B = ( — 3) ; 


Bol= ( See ) corresponds to dual feasible solution 


1/27/2 
ee) 
zg) by “Gago: 


The associated primal solution is feasible, i.e, B~'(2, 1)’ = (1,9/2)' > 0, 
hence optimality of the dual restricted master problem is achieved. 
3. The objective of the subprogram P (zr) is 
(= Fy S01, 123 eS 0 31 8) 
= (-1/2, 1/2, 3/2, —5/2, —4, 1/2, -1/2, 5/2). 


And P(z) can be decomposed into two programs, which are solved separately: 
(i) 

min —1/2x; + 1/2x2 + 3/2x3 — 5/2x4, 

s.t. xy +443 4+ x4 = 1, 


x2+%x3-x4=4, x7 =0, f= 1,...,4. 


(1) Take the basis and nonbasis B; = {1,2}, Ny = {3, 4}. Bt =I. 
The feasible basic solution is xz, = (1, 4)T>0, xn, = (0, oT. 


: oe =1/0 
y= B, cn = ( 2). 


T 
Spat Sule a Ae se SION > A DY. -, 3 
amen NONE (55) (i) | an) = haan) 
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The minimum reduced cost is min{3, —3/2} = —3/2, q =4. 
a4 = B,'(1,-1)? = (1,-1)7, p = 1. The minimum-ratio is min{1/1} = 
1, p=1. 


(2) Update the basis: 


af 
By = {4,2}, M. = (3, 1}. By! = © ‘ " (| "). 


The feasible basic solution is 
%e, = By 0,4) = 0,5)! 20) ay, = 00)5, 


“ = _t { —5/2 —2 
n= Biten =a" (“T)= (i). 


T 
eS nis, = ea) Ve (en >) = : 
ZN, = CN, ee (io) (2) = (sa): 


The minimum reduced cost is min{9, 3/2} > 0, and the optimal solution and 
associated objective value are 

(41, %2, %3,%4)" = (0,5,0, 1)", o1 = 1/2)5— 5/2 =0. 
(i1) 


min —4x5 + 1/2x6 — 1/2x7 + 5/2xs, 
s.t. x5 + 3x7 -—xg = 1, 
x6 +x7—-3xg=2, xj =O, f=5,...,8. 


Take the basis and nonbasis B; = {5,6}, N2 = {7, 8}. Be Sl, 
The feasible basic solution is xz, = (1, 2) = 0; Xn, = (0, 0). 


2 -4 
-T 
n= Biten=(j73). 


_ a, ype Sie ll 

ina en Me = (Sa) (13) (Ga) = (0): 

The minimum reduced cost is min{11,0} > 0, and the optimal solution and 
objective value are 

(%5, X6, X7,%8)' = (1,2,0,0)7, 2 = -44 (1/2)2 = -3. 

Summarizing (i) and (ii) gives an optimal vertex solution and optimal objective 
value to the subprogram below: 


u* = (0,5,0,1,1,2,0,0)', ¢* =¢) + =0+(-3) =-3. (9.35) 
4.() z= -3/2 > ¢* =-3. 


(u*)"H'™x = (0,5, 0,1, 1,2,0,0)(1, —3, —1, 1,2, 1, -1, —3)"™ = —10z. 
(u*)"c = (0, 5,0, 1, 1,2, 0, 0)(0, —1, 1, —2, —3, 1, -1, 1)? = —8. 
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Update the restricted master problem: 


Maxz,, 20 + g, 

s.t. —TIm+g<-5, 
Qn <1, 
—107 + g < -8, 
2r+g<M. (x) 


Iteration 3: 

1. Solve the restricted master problem. 
Column (—10, 1)! enters basis, g = 3. 
B-'(—10, 1)? = (1, —3/2)'. The minimum-ratio test: min{1/1} = 1, p = 1. 
The basis is updated as 


pa(7102) pii( 01) 
10 1/25 


corresponding to the dual feasible solution 


Cane, 


The associated primal solution is also feasible, i.e., B72, 1)! = (1, 6)T > 0. 
Thus optimality of the restricted master problem is achieved. 
3. The objective function of the subprogram P (zr) is 


(e= fay =] 0/2119 1 0 3 Fe) 
= (<1/2,1/2,3/,—572, 4,172, =172,4/)9, 


which is the same as that in Iteration 2, hence the optimal solution and associated 
objective value are given by (9.35). 

4. (2) g = ¢* = —3, optimality of the Benders master problem is achieved, and the 
optimal solution and objective value to the original problem are as follows: 


t=1/2, 37 =(67, 59) =(-2,1/2,-4,1/2), f= -2. 


9.6 Dual Benders Decomposition 


The decomposition presented in the preceding section handles the dual problem 
of a standard problem. It is also possible to handle a standard problem directly in 
a similar manner, resulting in a dual Benders decomposition. This section offers 
related results without detailed derivation. 
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Partition the variables of a problem into two parts, one of which are in linear 
form, i.e., 


Minny. f = f(w) + cox, 
s.t. F(w) + Ax = b, (9.36) 
wew,x>0, 
where A € R”™*", cE R", b € R”. Real-valued function f(z) and vector-valued 
function F(z) € R” and their domain W C R? satisfy certain conditions. If f(w) 
and F(w) are linear, and W = {w € R? | w => 0} holds, the preceding is clearly a 


standard LP problem. 
Viewing w as a parameter, introduce subprogram 


P(w) : min, c'x, 
st. Ax=b—F(w), x>0. 


The dual problem of the preceding is then 


D(w) : maxy ¢ = (b— F(w))'y, 
st. Aly<ce. 


Assume nonempty of the feasible region of D(w), i.e., 
Y={y | Aty<c} #9, 


whose vertex and extreme direction sets are denoted by 


Define the dual Benders master problem as follows: 


Miny,g f(w) + g, 

s.t. (u')' F(w) +g >(u')'b, ui €U, 
(v/)'F(w) > (v/)"b, vl eV, 
wewnry, 


(9.37) 


where 
T={w | F(w)+Ax =), x > O}. 


As for relationship between the dual Benders master problem and the original 
problem (9.36), we have the following result. 
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Theorem 9.6.1 (w, x) is an optimal solution to (9.36) if and only if x is an optimal 
solution to P(w) and (w, g) is an optimal solution to the dual Benders master 
problem. 


Thereby, one can solve problem (9.36) by handling the dual Benders master 
problem (9.37). Now turn to a relaxation problem yielded from dropping some 
constraints. Assume that subsets of the vertex and extreme direction sets of Y are 
known at some iteration, 1.e., 


Oe U0, Viv, 
Define dual Benders restricted master problem as follows: 


Miny,g f(w) +g, 

s.t. Gy Fw +e> G's, vw eu’, 
(v/)'F(w) > (v/)'b, vi EV’, 
wew. 


(9.38) 


Note A standby constraint f(w)+g => —M may be added to the preceding problem 
technically, where M is a number large enough. 


Theorem 9.6.2 Let (w, g) be an optimal solution to problem (9.38). 
If u* and ¢* are an optimal vertex solution and optimal value to subprogram 
D(w), then: 


(i) If g < ¢*, u* is anew vertex generated. 
(ii) If g = ¢*, (w, g) is an optimal solution to the dual Benders master problem. 


If D(w) is unbounded, then a new extreme direction is generated. 


Proposition 9.6.1 /f dual Benders restricted master problem is infeasible, so is 
problem (9.36). 


Algorithm 9.6.1 (Dual Benders Decomposition) Initial: dual Benders restricted 
master problem in form (9.38). This algorithm solves the restricted master problem 
(9.36). 


1. Call the simplex algorithm to solve (9.38). 

2. Stop if it is infeasible (the original problem is infeasible). 

3. If its optimal solution (w, g) is obtained, solve subprogram D(w). 

4. If optimal vertex solution u* and optimal value ¢* to D(w) are obtained, then: 


(1) Update (9.38) by adding the constraint, associated with u*, and go to step | 
ifg <¢*. 
(2) Stop if g = ¢*. 


The (w, x) is an optimal solution to (9.36), where x is complementary with 


ux. 


9.6 Dual Benders Decomposition 257 


5. If subprogram D(w) is unbounded, update (9.38) by adding the constraint 
associated with the new extreme direction. 
6. Go to step 1. 


We are interested in the linear case only. So, consider a special case of dual 
Benders decomposition: f(w), F(w) is a linear function. 
Assuming 


fwy=h'w, F(w)=Hw, W={we R"|w >=}, 
where H € R”*"!, h € R", then (9.36) becomes standard LP problem 


miny , f = h™w+c'x, 
s.t. Hw+Ax=b, (9.39) 
w,x > 0. 


Viewing w as a parameter, we handle the subprogram below: 


minyx f = cl x, 
s.t. Ax =b— Hu, (9.40) 
x > 0. 
For simplicity, assume that the set 
Y={yeR" | Aly <c} 49 (9.41) 
is bounded, whose vertex set is denoted by: 
OS Ge) cach (9.42) 


Thus, the dual Benders restricted master problem (9.38) becomes 


MiNy,¢ h™w + g, 
s.t. ')'Hwt+g>(u')'b, ui eU’. (9.43) 
w> 0. 


Assuming that there is an optimal solution (w, g) to the preceding problem, solve 
the following subprogram: 


D(w) : maxy ¢ = (b— Hw)'y, 
st. yey. 


If optimal value ¢* to D(w) is obtained, test for optimality of the original 
problem by the following criterion: 


gas". 
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Another way to derive the dual Benders decomposition is to use D-W decompo- 
sition and duality by dealing with the dual problem of (9.39): 


max b'y, 
s.t. Hly <h, (9.44) 
Aly <c. 


Let Y and U be defined by (9.41) and (9.42), respectively. Express y in a convex 
combination of vertices in Y. Then, the D-W master problem follows from (9.44), 
Le., 


AY 
max Soot u')ai, 


i=1 
Ss 


s.t. YS (ATu' a; <h, 


i=l 


(9.45) 


Thus, the dual Benders decomposition results from solving its dual problem, which 
is just the dual Benders restricted master problem (9.43). 

On the other hand, solving the dual problem of the D-W master problem of the 
original problem leads to the Benders decomposition. 

Finally, let us turn to the program with the staircase structure. We will not show 
that some of such programs can be solved by the Benders decomposition in a 
nested manner but show the so-called forward nested manner of the dual Benders 
decomposition by taking (9.16) again. 

Partition the variables into {x1} and {x2, x3, x4}. According to the dual Benders 
decomposition, we need to solve the subprogram with x! as its parameter, ice., 


min (c2)T x2 + (c3)Tx3 + (c4)T x4, 
S.t. Dox? =p? — Ay x!, 
A32x* + D33x? = b?, (9.46) 
Ag3x? + Dagx* = b+, 
x2, x3 x4 > 0. 


Thus, we turn to the dual problem of the preceding. In fact, it is easier to handle 
since only its objective function is related to parameter x!. In each iteration, the 
dual Benders restricted master problem is solved to provide a parameter value x! to 
the subprogram. Then (9.46) has a staircase structure similar to that of the original 
problem (9.16), with stairs reduced by 1. So, the dual Benders decomposition 
applies. Such a process is repeated until only one star is left, and the problem is 
easier to solve. 
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Chapter 10 M®) 
Interior-Point Method Ghost for 


As it is known, the simplex method moves on the underlying polyhedron, from 
vertex to adjacent vertex along edges, until attaining an optimal vertex unless the 
lower unboundedness is detected. Nevertheless, it could go through an exponential 
number of vertices of the polyhedron and even stay at a vertex forever because of 
cycling (Sect. 3.4). 

On the other hand, it is possible to achieve optimality directly, if one goes across 
the interior of the polyhedron. Indeed, there exists a ray, emanating from any point, 
to an optimal vertex, if any; in other words, a single iteration is enough to solve the 
LP problem if a “right” descent direction is known. Although unrealistic, this idea 
would be the motivation of the so-called interior-point method. 

To seek an optimal point, this method moves from interior point to interior point 
in the feasible region. Therefore, it can get rid of degeneracy that is closely related to 
vertices. Algorithms of this type mainly fall into three categories: potential function 
algorithms, affine algorithms, and path-following algorithms based on some log 
barrier function. This chapter is devoted to typical/efficient interior-point algo- 
rithms, such as the Karmarkar algorithm, affine interior-point algorithm, dual affine 
interior-point algorithm, path-following interior-point algorithm, including primal— 
dual algorithm, infeasible primal—dual algorithm, predictor—corrector primal—dual 
algorithm, and homogeneous and self-dual algorithm. 

Thereafter the following basic assumptions are made with the standard LP 
problem: 


Al: rank A =m. 

A2: Pt ={x € R"| Ax =b, x > O} FG. 

A3: Dt = {(y,z) € R™ x R"|Alty+z=c, z>O} FD. 
A4: c ¢ range A!. 


© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 261 
P.-Q. PAN, Linear Programming Computation, 
https://doi.org/10.1007/978-98 1-19-0147-8_10 


262 10 Interior-Point Method 


A4 is equivalent to the absence of a vector 7 such that c = A’, in which case the 
objective function is constant over the feasible region and vice versa if the feasible 
region is nonempty. Therefore, the optimality can only be attained on the boundary, 
if any. 


10.1. Karmarkar Algorithm 


The Karmarkar algorithm (1984) might be the most well-known interior-point 
algorithm, as it is of the polynomial-time complexity of the order lower than 
Khachiyan’s ellipsoid algorithm. the Karmarkar algorithm can be classified into the 
category of potential function methods. The basic idea behind it inspired the birth 
of some efficient algorithms of such type, changing the state of the art of LP to a 
larger extent. 
Consider the so-called Karmarkar standard problem: 
min f =c'x 
s.t. Ax =0, (10.1) 


ex=1, x>0, 


where A € R”*", cE R", rank A=m,m<n,n>2. 
As aconvex hull of 7 points e1, ... , én in the n-dimensional space, the polyhedral 


n 
T= ixeR’ aa xj =0 
j=l 


is a (n — 1)-dimensional regular simplex. It is noted that the n vertices of T° lie in 
symmetric positions, and its center is e/n. The radius of its inscribed sphere and 
circumscribed sphere are, respectively, 


r=l//na—1) <1, R=Vn—1/J/n <1. 


The feasible region of problem (10.1) is the intersection of simplex T’ and null 
space, {x € FR” | Ax = O}, of A. It is clear that there is an optimal solution if the 
feasible region is nonempty. 

Concerning problem (10.1), Karmarkar makes the following two assumptions: 


1. Ae = 0. So the center e/n of the simplex is an interior point of the feasible 
region. 
2. The optimal solution, say x*, satisfies c'x* = 0, i.e., the optimal value is 0. 


The preceding assumptions imply that c'x > 0 holds for all interior point x, in 
particular, the center e/n. 


10.1. Karmarkar Algorithm 263 


It is not difficult to convert a standard LP problem to the Karmarkar standard 
problem, we will not go into details though. Interested readers are referred to the 
literature, e.g., Tomlin (1987), Gay (1987), and de Ghellinck and Vial (1986). 


10.1.1 Projective Transformation 


Let x be an interior point, satisfying Ax = 0, e'x = 1 and x > 0. Denote by X the 
diagonal matrix, whose diagonals are components of x, i.e., 


X = diag(X1,..., Xn). 


Consider transformation 


vo Xt T(x) (10.2) 
or? aes : 
whose inverse transformation is 
pl) = 2 (10.3) 
x= x)= —-, . 
el Xx’ 


T (x), termed projective transformation, is a 1-1 mapping from T to I itself. In fact, 
for any x € T, it holds that x’ = T(x) € T; conversely, for any x’ € T, it holds 
that x = T~!(x’) € I. Under 7, in particular, each vertex e; (j = 1,...,n) of P 
corresponds to itself, and so does each edge; most importantly, x corresponds to the 
center e/n of I’, whereas any interior point corresponds to interior point. 

Using T, (10.1) is transformed to problem 


. — cl Xx! 
a eg sg 
s.t. AXx’ =0, (10.4) 


elx’=1, x'>0. 


which is no longer a LP problem though, because its objective function is not linear. 
However, it is known from (10.2) and e'x = 1 that when x is close to x, the 
denominator in the objective function can approximately be regarded as a positive 
constant, i.e., 
elx “ 
Tv. — = pees 
orks! = a = 1/)° xj/%j © 1/n. 
j=l 
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Karmarkar therefore employs c' Xx’ to replace the objective function approxi- 
mately. Precisely speaking, he used the following subproblem in each iteration: 


min f = cl Xx’, 
s.t. AXx’=0, (10.5) 
elx’=1, x'>0. 


The preceding and (10.4) have the same feasible region and initial interior point 
e/n. 


Proposition 10.1.1 Let x* be an optimal solution to problem (10.1). Then for 
any interior point x, there is an optimal solution (x*)' = T (x*), associated with 
objective value zero, to subproblem (10.5). 


Proof Note that for any interior point x > 0 to (10.1) and any feasible point x’ > 0 
to (10.4), it holds that 


n 
Tyv/ = TU tng : 
gan =) Xjx, 2e x min{x; | jf =1,...,n}>0. 


j=l 


It is known from (x*)’ = T (x*) and Assumption 2 that (x*)’ is an optimal solution 
to (10.4), satisfying clX (x*)’ = 0. In other words, there is a feasible solution (x*)’ 
to (10.5 with objective value zero. Assume that xX’ is its feasible solution with a 
negative objective value, i.e., c' XX’ < 0. Then it follows that 


aes 
= <9, 

el Xx! 

which contradicts that the optimal value of (10.4) is zero. Therefore, (x*)’ is an 
optimal solution to (10.5) with objective value zero. oO 


10.1.2. Karmarkar Algorithm 


Assume that a descent direction has been determined. If some components of x are 
close to zero, then the stepsize along the direction could be very small, and the 
associated improvement in objective function would be negligible. Problem (10.5) 
helps avoid this situation, to some extent, since x’s image, x’ = T(x) = e/n, in x’ 
space is now at the center of the simplex, the distance from which to each coordinate 
plane is the same. Of course, we do not really want to solve (10.5) itself, but only use 
it as a subproblem to determine a “good” search direction and associated stepsize. 
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To this end, denote the coefficient matrix of subproblem (10.5) by: 


AX 
F=| : |. (10.6) 


et 


Then the orthogonal projection matrix from x’ space to the null of F is 
P=1-—F'(FF')'F. (10.7) 
Thus, the orthogonal projection of the objective gradient Xc is 
Ax = PXc=(1— F'(FF')"'F)Xc. (10.8) 


Proposition 10.1.2 Vector Ax is nonzero, satisfying FAx = 0 and (Xc)' Ax > 0. 


Proof Assume Ax = 0. It is known from (10.8) that there ish € R”, hy41 such 
that 


0S FW he. 
Therefore, for any feasible solution x’ > 0 to subproblem (10.5), it holds that 
cl Xx! = (h", hg) Fx! = A AX’ + hmyielx! = hms. 


In other words, the feasible value is constant. Note that this is the case for xX ; 
constructed from any interior point x. In particular, for x = e/n, subproblem’s 
objective value at the center e/n of the simplex is c'e/n*; on the other hand, it is 
known from Proposition 10.1.1 that the optimal value is zero. Hence it follows that 
cle/n* = 0, which implies that e/n is an optimal solution, contradicting that any 
interior point is not optimal. Therefore, Ax 4 0. Finally, from (10.8) and P? = P 
together with P! = P, both FAx = Oand (Xc)T Ax > 0 follow. oO 


The preceding Proposition says that —Ax # 0 is a descent feasible direction. 
Further, it is not difficult to show that — Ax, which is within the null of F’, forms the 
maximum possible angle with the objective gradient, that is, — Ax is the steepest 
descent feasible vector in the null. As a result, if the stepsize from the center along 
the direction is less than the radius of the inscribed sphere of the simplex, then the 
new interior point in x’ space must lie in the interior of the feasible region, hopefully 
leading to a satisfactory decrease in the objective value. More precisely, the new 
interior point in x’ space is determined by: 


# == —ap—— (10.9) 


266 10 Interior-Point Method 


where a € (0,1), po € (0,r], and r is the radius of the inscribed sphere. Thus 
mapping it to the original x-space via the inverse transformation gives 


KE = XK Jel XR. (10.10) 
Then, an iteration is complete. 


The overall steps are summarized into the following algorithm. 


Algorithm 10.1.1 (Karmarkar Algorithm) Given tolerance € > 0. Initial: k = 
1, x =e/n. 
This algorithm solves the problem (10.1). 


1. Compute Ax = PXc = (I — F'(FF')~'F)Xc, where F is defined by (10.6). 
2. Compute *’ = e/n — ap Ax/||Ax||. 

3. Compute * = Xx! /eT XX’. 

4. Stop if cTx < . 

5. Setk=k+1. 

6. Go to step 1. 


The Karmarkar algorithm’s inspiration is perhaps more valuable than itself, 
which stimulated coming out of some very successful algorithms of practical 
use. The idea that determining a search direction in the image space and then 
transforming it back to the original space has influenced the development of 
interior-point methods greatly. At present, search direction of a typical interior-point 
method is usually a combination of a descent direction, stemming from the negative 
gradient, and a centering direction, generated via some transformation. Such a 
combination, which allows the algorithm to go farther in each iteration, is the real 
cause of the success of the interior-point methods. As a “by-product,” in addition, 
approaching an optimal solution by a sequence of interior points overcomes the 
troublesome degeneracy encountered by the simplex method. 


10.1.3 Convergence 


To analyze Algorithm 10.1.1, define the following potential function over T of 
problem (10.1): 


n n T 
F(x) = f(x,c) Sninclx) — Yo inx; = Doin. (10.11) 
7 
jal “J 


j=! 


The corresponding potential function over I’ of problem (10.5) is then 


(10.12) 


tp VK A loo Ty “ / - ot Xa! 
f(x’) = fr’, Xc) =nIn(c Xx')— ) nx, = Doin a 
j=l J 


j=l 
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Proposition 10.1.3 The potential function values at any point x € TY and at its 
image x’ differ by a same constant that is dependent on x. 


Proof From (10.3), (10.11), and (10.12), it follows that 


cl Xx’ 


fo =fT'@’) = yo in wa In(IT_ Xj) = f(x) - In (ITs _) xj). 


j=l j 
Oo 


It is therefore seen that if a x’ is determined such that f’(x’) is less than f’(e/n) 
by a constant, then the corresponding f(*) is less than f(x) by the same amount. 
We will estimate 


n 
f'@!) = nin(cT X28") — Yn. (10.13) 
j=1 


Lemma 10.1.1 Jf Ax and x’ are, respectively, determined by (10.8) and (10.9), then 
it holds that 


cl Xe 


Inc! Xx’ < In —ap. (10.14) 


Proof From P? = P, P™ = P and (10.8), it follows that 
chX Ax =c!XPXc =c!XP*Xc = (PXc)'(PXc) = ||Ax|* > 0, (10.15) 
which together with (10.9) gives 


Ty Ty Ty 
— x c XA cx 
xt = agp eptAa, (10.16) 
n || Ax || n 


Now consider 
min f = c'Xx', 


st. AXx’ =0, 
efx’ =1, 


Ix’ —e/n|| < R, 


(10.17) 


where R is the radius of the circumscribed sphere, i.e., 


R=Jn—I/Jn <1. 
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Noting that — Ax is the orthogonal projection of the negative objective gradient onto 
the null of /, and 


{x ER" | lx’ —e/nll < R} 


is the spherical domain with radius R and the simplex’s center as its center, it is not 
difficult to show that the optimal solution is 


: e Ax 
x (R)=-—R é 
n || Ax| 


On the other hand, since ||x’ — e/n|| < R is the circumscribed sphere of I, 
the feasible region of (10.17) includes the feasible region of subproblem (10.5), 
therefore the optimal value of the former is no more than the optimal value zero of 
the latter. So, considering (10.15) leads to 


Ts clXe 
c Xx’ (R) = —— — R||Ax|| <0, 
n 


or 


c'Xe 
—||Ax|| < -——., 
nR 


which, combined with (10.16), R < 1, and clx > 0, gives 


TXe cl Xe 
ap 


=, Cc 
cl XR < - 
nR 


exe cl Xe 
= (1 — ap/R)—— < (1—ap)——. 
n n 
Consequently, taking logarithm of the two sides of the preceding together with 
In(1 — ap) < —ap 


leads to (10.14). oO 


Lemma 10.1.2 [fx’ € T satisfies \|x’ — e/n|| < ap, then 


. ’ - (nap)? 
7 2%) Bim Ze 4. Td ana (10.18) 


Proof Since ||x' — e/n|| < ap, it holds that 


I/n—ap <x <1/n+ap, J HApeesgn, 
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or 
1 —anp < nx; <1+anp, 7 =1)csa5n. (10.19) 
Based on Taylor formula, In(nx’;) can be expressed as 
In(nx’,) = Ind + nx’, — 1) — (nxi, — 1)°/(267), 


where 6; is between 1 and nx’., hence it is seen from (10.19) that 6; > 1 — anp. 
Therefore, the following nea holds: 


; ; (nx', — 1)? 
In addition, it is known from the assumption that 
n 
Yo(nx’ —1)=ne'x'-n=0 
j=l 
and 
n 
Dina — 1)? = Inx! — elf? =n? Ilx! — e/n? < (anp)’. 
j=l 
Then, it follows from (10.20) that 
2 
y In(nx',) > =(onp) 
20 — 2(1 — anp)2’ 
which implies (10.18). oO 


Theorem 10.1.3 [fp = 1/n <r, a € (0, 1/2), then the complexity of the number 
of iterations required by the Karmarkar algorithm is O(nL), where L is the binary 
digits of all input data. 


Proof Combining (10.13), (10.14), and (10.18) gives 


2 
ano — Sind) + 5 One = f’(e/n) —48(n, p, a), 


f'@) < nin’ 
= 2(1 


(10.21) 


where a lower bound of the decrement of the potential is 


(anp)? 


b(n, p,a) = anp — 20 —anp)?” 
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Substituting o = 1/n into the preceding gives a function, only related to a, i.e., 


7 a  — a(a — 2)(a — 1/2) 


for which it is clear that 


>0, 0<a <1/2, 
d(a)}=0, a=1/2, 
<0, 1/2<a<l. 


Therefore, from the assumption of this Proposition, it is known that d(a@) > 0. 


Let x be the original image of x’, determined by (10.10). According to Proposi- 
tion 10.1.3, there is a constant jz related to x such that 


f@=fe/n+u, f@O=f'@)+u. 
The preceding together with (10.21) gives 
f (&) — f(&) = f'@) — f'(e/n) < -6@), 
and hence, for each iteration, it holds that 
IQ)s FE) 8); 


It is noted that the initial interior point is e/n. Therefore, if X is the interior point, 
generated at iteration k = 1, 2,... (step 3 of Algorithm 10.1.1), then 


f(k) < f(e/n) — kd(@). 
By (10.11), therefore, it is valid that 
n n 
nin(c'x) — x Int; <nIn(cle/n) — ~ In(1/n) — kd(a), 
j=l j=l 


n 
from which, x € T and function > In x; attaining the highest value at e/n over I, 
j=l 
it follows that 


nin(c'£) < nIn(cle/n) — kd(a). 
The preceding is equivalent to 


chk < ec BO/"(CTe/n). (10.23) 
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Let € be a given tolerance on the optimal value. Then it is clear that there isa t > 0 
such that 


E 


cle/n 


ate 2 (10.24) 


Once the number k of iterations attains the smallest integer such that 
k > (t/5(a))nL, 


then considering (10.24) gives 


€ 


gE 2g eth ee 7 : 
cre/n 


which and (10.23) together lead to 


clR <e. 


Under the assumption, therefore, the complexity of the number of iterations, 
required by the Karmarkar algorithm, is O(nL). oO 


Since the computational complexity of a single iteration is O(n*), the overall 
computational complexity of the Karmarkar algorithm is O(n*L). Further, Kar- 
markar improves the complexity of a single iteration to O(n>), hence the overall 
complexity reduces to O(n3°L). 

In principle, it is desirable to have a larger lower bound on the decrement of 
the potential function. To this end, setting the derivative of (10.22) to 0 leads to its 
maximum a* * 0.3177 with value 6(a@)* ~ 0.2093. Therefore, taking a = 0.3177 
in the Karmarkar algorithm appears to be a good choice. However, it is not the case 
in practice, experience shows that an a close to | often accelerates convergence, 
even if the polynomial complexity cannot be guaranteed when a > 1/2, in theory. 


10.2 Affine Interior-Point Algorithm 


The Karmarkar algorithm is not very convenient for practical use since it is only 
amenable to the Karmarkar standard problem. To handle the standard LP problem 
directly, the so-called affine algorithm was soon proposed as a variant of the 
Karmarkar algorithm (Barnes 1986; Cavalier and Soyster 1985; Karmarkar and 
Ramakrishnan 1985; Vanderbei et al. 1986). However, it is immediately found that 
the same algorithm was proposed by Dikin as early as in 1967 (without convergence 
proof), but not noted by the academic community, unfortunately. 

Thereafter, we are concerned with the pair of the standard LP problems (5.1) and 
(5.2). 
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10.2.1 Formulation of the Algorithm 


Let X be the current interior point. Again denote by X the diagonal matrix, whose 
diagonals are components of x. It is clear that affine transformation 


x’ =X !x 


maps the positive octant to itself, and x to its center e. This transformation turns 
problem (5.1) into 


sy oT Vy! 
mt Ads =, x'20 (00.25) 
The orthogonal projection from x’ space onto the null of AX is 
Para RA (Ax At AX; (10.26) 
and the orthogonal projection of objective gradient Xc is then 
Ax’ = PXc = (I — XA'(AX*A')“!AX) Xe. (10.27) 


Lemma 10.2.1 Under the basic assumption A4, it holds that Ax' # 0. 


Proof Equation (10.27) can be written as 
Ax’ = X(c— A’), 
where 
¥ = (AX*Al) TARR. 


Thus Ax’ = 0 implies c = Ay, which contradicts assumption A4. Therefore 
Ax’ £0. Oo 


To determine a new interior point in x’ space, take — Ax’ as search direction: 
x’ =e — pwAx'/||Ax’ |. 


If uw € (0, 1) is taken, then xX’ lies within interior of the sphere with radius | at e. 
The original image of x’ in x space is then 


£ = — pXAx'/|Ax'|, 


which is the short stepsize iteration scheme of the affine algorithm. 
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It is verified that the point 
R= x—-aXAx’ (10.28) 


satisfies Ax = b for any a. In particular, for the short stepsize a = u/||Ax’|| > 0, 
it holds that x > 0, hence the new iterate is again an interior point. Combing P = 
P?, P™ = P, and (10.27) leads to 


cht =clx —ac'XPXc = cx — al[Ax' |? < c'x, (10.29) 


which implies that the objective value strictly decreases. 

It is desirable to go further without violating constraints. It turns out that letting 
the new interior point closer to the boundary leads to a bigger decrement of the 
objective value and significantly enhances the efficiency of the algorithm. To this 
end, the following result should be noted first. 


Lemma 10.2.2 If Ax’ < 0, the standard LP problem (5.1) is unbounded below. 


Proof Under the assumption, < defined by (10.28) is clearly an interior point for 
any a > 0. Itis seen from (10.29) that 


cl = cl¥ — alJAx’ |? > —00 (a > 00). 


Therefore, the original problem is unbounded below. oO 
Assume now that Ax’ < 0. In this case, the maximum stepsize attaining the 
boundary is 1/ max(Ax’), leading to the following “long stepsize” scheme: 


% =X —AXAx!'/ max(Ax’), (10.30) 


where A € (0, 1) is a stepsize. 

Accompanying the original interior point x, a dual estimate (y, z) is expected. 
If ignoring the nonnegative requirement for z but fulfilling the complementarity 
condition as much as possible, (y, z) should be the solution to the following least- 
squares problem: 


min (1/2)||Xz|l?, 
st. z=c—Aly. 


It is not difficult to solve the preceding, leading to 
y = (AX7Al)!AX?c, z= X7!PXc. (10.31) 


Thus (10.27) can be written alternatively as 


Ax’ == X(c — A'y) = Xz, (10.32) 
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which reveals the relation between the search direction and the dual estimate. It is 
noted that XZ is just the associated dual gap (see Sect. 5.3). 

The overall steps with long stepsize are put into the following algorithm (without 
generating dual estimates). It is known from (10.32) that the dual gap tens to zero as 
the procedure converges. Therefore, optimality is achieved approximately whenever 
|| Ax’ || becomes small enough. 


Algorithm 10.2.1 (Affine Interior-Point Algorithm) Given A € (0,1), € > 0. 
Initial: interior point x > 0. 
This algorithm solves the standard LP problem (5.1). 


. ComputeAx’ = PXc = (I — XA'(AX7A!)“!AX)Xc. 
. Stop if || Ax’|| < € (optimality achieved approximately). 
. Stop if Ax’ < 0 (unbounded below). 

. Update by ¥ = ¥ — AX Ax'/ max(Ax’). 

. Go to step 1. 


nA WN Re 


It is seen from (10.27) that the algorithm’s major work in each iteration is the 
computation of projection Ax’, concerning the solution of m x m system below: 


(AX? Al)y = AX7c. 


In contrast to the Karmarkar algorithm, the affine algorithm is simpler and yet 
performs better. In fact, the latter is the earliest interior-point algorithm found to 
possibly outperform the simplex method. 


10.2.2. Convergence and Starting-Up 


Firstly, we state two convergence results without proof. The first, given by Dikin 
(1974), is under the nondegeneracy assumption (see also Gonzaga 1990; Vanderbei 
and Lagarias 1990). 


Theorem 10.2.3 Assume the existence of an optimal solution to the standard 
problem (5.1). If all feasible solutions are nondegenerate, then the sequence of 
interior points, generated by (10.30) (A € (0, 1)), and sequence of dual estimates, 
generated by (10.31), respectively, converge to relative interior points of the primal 
and dual optimal faces. 


If the range of 4 is restricted properly, the long stepsize affine algorithm 
converges without the nondegeneracy assumption, as the following theorem says 
(Tsuchiya and Muramatsu 1995). 


Theorem 10.2.4 Assume the existence of an optimal solution to the standard 
problem (5.1). Then the sequence of interior points, generated by (10.30) (A € 
(0, 2/3]), converges to a relative interior point of the primal optimal face, and the 
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sequence of dual estimates, generated by (10.31), converges to the relative analytic 
center of the dual optimal face, with asymptotic descent rate | — i for the objective 
value. 


When 2 > 2/3, however, the convergence of the affine algorithm is not 
guaranteed. Counterexamples have been found, showing that 2/3 is the maximum 
value of A for ensuring convergence (Hall and Vanderbei 1993). As for practice, 
nevertheless, the case is just the contrary: it is faster and more reliable to take a A 
value close to | (e.g., A € [0.9, 0.99]). On the other hand, it is shown that even if 
X. < 2/3, the algorithm solves the Klee—Minty problem by traversing neighborhoods 
of all the 2” vertices and hence is not of polynomial complexity (Megiddo and Shub 
1989). 

Now turn to Phase-1 to provide an initial interior point to the affine algorithm. 

Given any n-dimensional vector h > 0 such that b— Ah 4 0 (h = e seems to be 
a good choice). Introducing an artificial variable x,+41 and a normalized vector 


dn41 = (b— Ah)/\|b — Ahll, (10.33) 
construct the following auxiliary problem: 


min Xp41, 
10.34 
St. Ax + an4i%n41 =b, xX,Xn41 = 0. ( ) 


There is an optimal solution to the preceding problem. Moreover, x = h, Xn41 = 
||b—Ah|| is an interior-point solution available. If the optimal value is strictly greater 
than 0, then the original problem is infeasible. 

Assume that solving the auxiliary problem by the affine algorithm with A ¢€ 
(0, 2/3] produces a sequence of interior points converging to x with objective 
value 0. Thus, x°° is a relative interior point of the feasible region of the original 
problem. 

Define 


B={je¢A|x* > 0}, N= {je A|x? = 0}. 
Then, the original problem reduces to 


min Clee; 
st. Bxp=b, x>0, 


which has an interior point x > 0 available. Then, the affine algorithm can get 
itself started. 
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10.3 Dual Affine Interior-Point Algorithm 


The basic idea behind the affine method is applied to the dual problem to derive its 
dual counterpart (Andersen et al. 1996). 

Consider the dual problem (5.2). Assume that (y, Z) is the current interior point, 
satisfying dual constraints. Denote by Z the diagonal matrix with components of Z 
as its diagonals. The affine transformation z = Zz’ turns the dual problem to 


max b'y, 
st. ATy+Zz=c, 7 >0 


or equivalently, 


max b'y, 


if = 10. 
st. ZATy+2/=Z-'e, 7¢>0. eo) 


Note that the affine transformation z = Zz’ maps the positive octant of z-space to 
itself, Z to its center e. Since A has full row rank, Z —! AT has full column rank. 
Assume that the QR factorization of Z~! AT is 


Z-'AT =[Q), Oo] (5) = O1R, (10.36) 


where [Q1, Q2] is orthogonal, Q; € R”*” and Q2 « R’™*"—™, and R e R™™*™ 
are upper triangular. 
Substituting (10.36) into the equality constraint of (10.35) gives 


O\Ry+z = Zé 


Premultiplying the preceding by R~! Q?, noting O10) = I, and rearranging lead 
to 


y= R'Q'z-le - Rote’, 
Combining the preceding and 
QQ, +Q207=1, O30.=1, 
we convert problem (10.35) into the following standard form with respect to z’: 


: Tp-lot./ 
min b'R™'Q;z’, 


te 10.37 
s.t. Qtz’ = O17 *c, Zz >O0, ( ) 


where the constant —b' R~! O1Z ~l¢ was omitted from the objective function. 
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Proposition 10.3.1 The orthogonal projection of the objective gradient of (10.37) 
to the null space of Q3 is equal to 


Az = Z~!AT(AZ~7A")!b. (10.38) 


Proof By Q3 Q, = 0, the orthogonal projection of the objective gradient to the null 
becomes 


Az’ = (I — Q2(0302)'Q3)OQ1R ‘b= QR 'b. 
By ora, = I, the preceding is equal to 
Az’ = Q\(RR7')R-"b = Qi R(R'R)|'b = O;R(R™Q] OR) |b, 


which together with (10.36) gives (10.38). oO 


Thereby, —Az’ is a favorable search direction in z'-space. For a new iterate to 
satisfy the equality constraint of (10.35), Az’ and the associated search direction 
Ay in y-space should fulfill the condition 

Z'ATAy + Az’ =0, 
premultiplying which by AZ~! and using (10.38) gives 
Ay = —(AZ~*A!)!AZ! Az’ = —(AZ~7 Al) |b. (10.39) 
Via z = Zz’, vector 
Az’ =—-Z~'!ATAy 
can be transformed back to z-space, 1.e., 
Az=—Al'Ay. (10.40) 

The long stepsize line search scheme is then 

j=y—ABAy, 7=Z-ABAz, (10.41) 
where 4 € (0, 1), and 
6 = min{z;/Az; | Azj > 0,7 =1,...,n}. (10.42) 


In the case when Az < 0, the preceding scheme is not well-defined, and the dual 
problem is unbounded, or the primal problem is infeasible. 
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As for an estimate, say x, of the associated primal solution, it can be determined 
as follows. If ignoring the nonnegative requirements but attempting to satisfy 
complementarity condition, the x should solve the following least-squares problem: 


min ||Zx|l, 
st. AZ—!(Zx) =b. 


The solution to which can be obtained by taking a look at the minimum 2-norm 
solution to AZ~!w = b and (10.39), ice., 


% = Z~*AT(AZ~*A")“|b = Z~7 Az. 


Consequently, when x > 0 and dual gap c'x — b'y becomes sufficiently small, the 
procedure, generating the sequence of dual interior points, can be terminated. 
The overall steps are put into the following algorithm. 


Algorithm 10.3.1 (Dual Affine Algorithm) Given 4 € (0,1), € > 0. Initial: 
interior point (y, Z). 
This algorithm solves the standard LP problem (5.1). 


. Compute (Ay, Az) by (10.39) and (10.40). 

. Stop if Az < 0 (dual unbounded or primal infeasible). 

. Compute x = Z~* Az. 

. Stop if x > 0 and cx — b'y < € (optimality achieved approximately). 
. Update (y, z) by (10.41) and (10.42). 

. Go to step 1. 


NANNBWN 


The preceding algorithm starts from a dual interior point. (y = 0,z = c) is 
available one in case of c > 0. Otherwise, an approach similar to that described in 
the last half of Sect. 10.2.2 can be employed to generate an initial dual interior point 
by solving the following auxiliary problem: 


max —Ym-+1; 


s.t. [aT ic—el( y )te=c Z, Ynt1 = 0. 
Ym+1 


The preceding can be handled by the dual affine algorithm since it is upper bounded 
and has an interior point available, i.e., 


y=0, Yntt1=1; Z=e. 
We point out that primal and dual affine algorithms can be derived from each 


other and share similar convergence properties (Tsuchiya 1992), but the latter 
performs better in practice. 
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10.4 Path-Following Interior-Point Algorithm 


These types of methods perform very well in practice. Such a method can be 
regarded as one using a homotopy strategy, in which one creates a trivial problem, 
compared with a hard problem to be solved, so that there is a continuum of problems 
between them and handles backward from the trivial problem to the hard problem 
by solving the problems in between (exactly or approximately). 

Taking the standard LP problem (5.1) as the hard problem, we construct the 
trivial problem, whose optimal solution is an interior point. 

As the border of the positive octant, coordinate planes x; = 0, j = 1,...,n, 
defining the scope of the feasible region, might well be deemed as a “wall.” 
We have seen that the Karmarkar algorithm and/or the affine algorithm employ 
transformations to create a “centering force” to push the iterate leaving the wall. 
The combination of the centering force and the pushing force along the direction of 
the negative objective gradient gives a favorable feasible descent search direction. 
Another approach for generating “centering force” is via logarithmic barrier func- 
tion, which was initially employed by Frisch (1955) to solve nonlinear problems. 
After the Karmarkar algorithm emerged, the logarithmic barrier function draws 
high attention from the community and is applied in the path-following method 
to construct the trivial problem. 

As nonnegative constraints x > O are inactive for interior points, we remove 
them and add logarithmic barrier terms to the objective function to construct the 
problem below: 


(P,,) min f(x) =clx — pw day Aas 


st. Ax =b, 


where pz > 0 is called barrier parameter. Implying x > 0, (10.43) characterizes a 
class of problems with parameter jz > 0. Because of the barrier terms, f — +00 
as x; — Ot, so that minimization of f “bounces” points from the “wall” to prevent 
them from leaving the interior of the feasible region. 

Related to the hard problem by yp, the trivial problem (10.43) is nonlinear but 
not difficult to handle. The gradient and Hessian matrix of the objective f(x) are, 
respectively, 


Vi@)=c—px es, VP@)=ux *, 


where X = diag(x1,...,X,). It is clear that f(x) is a strict convex function over 
region x > 0. The Lagrange function of problem (P,,) is then 


n 
L(x, y) =c!x — wy? Inx; — y'(Ax — b). 
j=l 
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Therefore, the point x > 0 is an optimal solution to (P,,) if and only if there is a 
y € R” such that 


Aceh = 0. (10.44) 
c—pX'e— AT y=0. (10.45) 


Using the notation 
Z=<¢c= Aty, Z = diag(z1,.-.-, Zn); 


the preceding condition becomes the following system in variables x, y, z: 


Ax =b, (10.46) 
Aly+ z=c, (10.47) 
Xz =e. (10.48) 


Megiddo (1989) shows that: 


(i) For each yz > 0, problem (P,,) either has an unique optimal solution or is lower 
unbounded. 

(ii) If there exists a solution x > 0 to the system for some yz > O, then so is a 
solution x (jz) to the system for all w > O, and x (jz) is a continuous curve. There 
is a limit lim x(jz) as 44 > O*, which is an optimal solution to the problem (P). 


If the interior of the feasible region of (P) is nonempty and lower bounded, 
(10.46) ~ (10.48) determine a unique path x (jz) to an optimal solution. Such a x (jz) 
is termed central path or trajectory. The common idea of various path-following 
algorithms is to find an optimal solution by following the path approximately. 

On the other hand, the central path can also be derived in another way. We can 
handle the standard LP problem directly via the optimality condition (5.15), viewed 
as a nonlinear system of x, y, z subject to x,z > 0. To generate interior points 
across the interior of the feasible region and approach an optimal solution on the 
boundary, it is only possible to fulfill the complementarity condition approximately 
by replacing Xz = O with Xz = v, where v > O is a parameter vector. The 
associated solution x(v) to the system is expected to tend to an optimal solution 
as v tends to 0. In view of the symmetry of x;z; = vj, Vj =1,...,n, itis natural 
to allow components of v to be equal by setting v = je, which leads to (10.46) 
~(10.48) consequently. So, the “hard” problem is the optimality condition, while 
the trivial problem is system (10.46—10.48). Of course, it is not realistic to solve all 
systems for continuous jz, but for some sequence {¢} instead, as will be seen in the 
next section. 
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A realization of the path-following strategy is to design a monotone descent 
sequence {2%} with limit 0, and for every fixed parameter uw = ux, k = 1,..., solve 
system (10.46)~(10.48) for its solution x (zx). In general, it is difficult to directly 
obtain the exact solution to the nonlinear system (10.48). Instead, we employ the 
Newton method to find an approximate one. It is unnecessary to pay a high price for 
the accuracy since what we are interested in is not the solutions themselves but their 
limit. 

Thus, only a single Newton step is taken to obtain an approximate solution in 
each iteration, and the monotone descent jz, are generated one by one. 

Assume that the interior point (x, y, Z) satisfies (10.46) and (10.47) and approx- 
imately satisfies (10.48). We need to determine (—Ax,—Ay, —Az) such that 
(x — Ax, y — Ay, z — Az) is a new approximate solution, which satisfies 
(10.46)~(10.48), Le., 


A(x — Ax) = Bb, 
Al(y—Ay)+@ — Az)=e, 
(X — AX)(Z— Az) = ne. 


The preceding expressions are equivalent to 


AAx = Ax—b, (10.49) 

AlAy+ Az=Aly+Z-—c, (10.50) 

ZAx+XAz = XZ+ AXAz-— ue, (10.51) 

where AX = diag (Ax), ..., Ax,). Itis known from the assumptions that the right- 


hand sides of (10.49) and (10.50) are equal to zero. Then, omitting the second-order 
term AX Az of the right-hand side of (10.51) gives the so-called Newton equation: 


AAx = 0, (10.52) 
A’Ay+ Az =0, (10.53) 
ZAx+XAz = XZ—upe. (10.54) 
Introduce notation 
D=Z"'X. (10.55) 


Premultiplying (10.54) by AZ~! and noting (10.52) lead to 


ADAz = ADz — wAZ~!e, 
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and premultiplying (10.53) by AZ -1¥ gives 
ADA' Ay + ADAz = 0. 
From the preceding two equations, it follows that 
Ay = —(ADA')!A(Dz — wZ~!e). (10.56) 
By (10.53) and (10.54), on the other hand, it holds that 


Az = —ATA y, 
Ax = D(Z —Az) — Zoe. 
The preceding three expressions together determine the so-called Newton direction. 
Considering that the Newton point, determined along the direction by taking 
stepsize | probably violates the nonnegative conditions, the new iterate is defined as 


k=x-aAx, y=y-aAy, Z7=Z-adz, (10.57) 


with the stepsize a determined by: 


a =Amin{ap, ag}, (10.58) 
where A € (0, 1),! and 
@p =min{x;/Ax; | Axj>0, j=1,...,n}, (10.59) 
oa =min{z;/Az; | Azj>0, j=1,...,n}. : 


Note that the new interior point satisfies conditions (10.46) and (10.47), but only 
approximately satisfies (10.48). 

Since the iterate is primal and dual feasible, the iteration process can be termi- 
nated when the dual gap becomes sufficiently small, e.g., by using the following 
criterion: 


x1Z/(1+|b'y]) <€, 


where 0 < € < | is the precision tolerance. The end iterate is then regarded as an 
approximate optimal solution. 

As a key factor, the barrier parameter jz affects the behavior of the algorithm 
greatly, as it determines the magnitude of the centripetal force, compared with that 
of the “pushing down force.” It turns out that iterates too close to the central path 


') = 0.99995 may be a suitable choice—a long stepsize performs better than a short stepsize. 
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are unfavorable and degrade computational efficiency. For each given ju, it is only 
needed to go by one step along the Newton direction. As the larger the yw, the greater 
the rebound to the point, so it is necessary to reduce jz gradually as iterates approach 
an optimal solution. It is difficult to find an “ideal” way to do so, however. In this 
aspect, extensive research has been done, and schemes have been suggested. Some 
schemes make the algorithm have polynomial time complexity (e.g., Kojima et al. 
1989; Jansen et al. 1996 offered a unified form). 

The first practicable scheme, proposed by McShane et al. (1989), monotonically 
reduces jz with the varying dual gap in the solution process. 

From (10.57), it is follows that 


So, the decrement of the dual gap is 

5 = a(c! Ax — bTAy). (10.60) 
It is known by A(x — Ax) = Db that 

bY(5 — Ay) = — Ax)"AT(— Ay), 
and by A'(y — Ay) + (Z— Az) = c that 
cl (% — Ax) = (y — Ay) "A(& — Ax) + @ — Az) — Ax). 

From the preceding two equalities, it follows that 
c'(¥— Ax) —b'(y— Ay) = (Z—Az)'(#— Ax) = ZX Az ¥—Z7Ax+ Az! Ax. 
In addition, by (10.54) it holds that 


ziAx +x1Az=x! 


Z—Nnw. 
Combining the preceding two equalities gives 
cl(x — Ax) - by — Ay)=nuw+ Az Ax, 
which, by omitting the second-order term, can be written as 
“(AGS b™ Ay) mw —clx +b 5 4+np. 


Substituting the preceding to (10.60) leads to 


8X a(clx —b'y —np). 
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In order for dual gap’s decrement to be positive, therefore, jz should satisfy 
we < (cl# — bTy)/n, 


at least. On the other hand, too big jz is not suitable, since it may cause numerical 
instability. In this aspect, one might take 


w= (c'x — b 5) /n?. 


Overall steps are summarized into the following algorithm. 


Algorithm 10.4.1 (Primal—Dual Interior-Point Algorithm) Given 0 < « < 
1, A € (0, 1). Initial: interior point (x, y, z). This algorithm solves the standard 
LP problem. 


. Stop if 7Z/(1 + |b’ y|) < € (optimality achieved approximately). 

. Compute w = (c'x — bTy)/n?. 

. Solve system (ADA')Ay = —A(Dz — Z7!e). 

. Compute Az = —ATAy, Ax = D(Z — Az) — uZo!e. 

. Update y, z, x by (10.57), where a is determined by (10.58) and (10.59). 
. Go to step 1. 
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10.4.2 Infeasible Primal-—Dual Algorithm 


Since it was not easy to obtain an initial interior point to get it started, the use of the 
primal—dual method was limited, as prompted a simple variant to appear. 

Terms infeasible interior point and interior point differ only from that the 
former does not necessarily satisfy primal and dual equality constraints. To get 
started from an infeasible interior point, Lustig (1990) and Tanabe (1990) modify the 
primal—dual method presented in the previous section. The basic idea is similar to 
that behind the primal—dual simplex method (Sect. 6.2) but is realized in a different 
framework. 

Assume that the current iterate (x, y, z) be an infeasible interior point, satisfying 
x, Z > O but the residuals 


are not equal to 0, in general. 
Omitting the second-order term AX Az from (10.49)~(10.51) gives the Newton 
equation, i.e., 


AAx = rp, (10.61) 
AlAy+ Az=ra, (10.62) 
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ZAx+XAz = XZ-—ue. (10.63) 


Using notation D defined by (10.55), the solution to Newton equation can be written 
as 


Ay = (ADA') 1 (ry — AD(Z — ra) + BAZ), (10.64) 
Az=rq—A'Ay, (10.65) 
Ax = D(z — Az) — uZo!e. (10.66) 


In fact, the last two expressions can be derived from (10.62) and (10.63). In addition, 
premultiplying (10.63) by AZ~! and noting (10.61) give 


AZ 'XAz=-—r, + AZ '(Xz— ne), 
and premultiplying (10.62) by AZ~!X leads to 
AZ |X ATAy+ AZ! XK Az = AZ Xrq. 
Then, (10.64) follows from the above two equations. 

Except for different search directions, determination of the new iterate is the 
same as that for the primal—dual method. The infeasible primal—dual method also 
uses updating formula (10.57), where stepsize a is determined by (10.58)~(10.59), 
with A € (0, 1) (usually, A = 0.99995 is taken). 

The barrier parameter is determined by Lustig et al. (1991): 

w= (cx —b'¥+ My)/o(n), (10.67) 


where 


y = ||AX— bl|/ ||Ax° — bl| + |ATY + Z—cell/I[ATy® + 2° — cl], (10.68) 


M = &$(n) max{max |cj|, max |bj]}, (10.69) 
j= i= 
n2, Ifn < 5000 
— : _ ; 10.7 
o@) en Ifn > 5000. owen 


Initial (x°, y°, z°) and & are determined by another algorithm (see discussions 
following Algorithm 10.4.3). 
Overall steps are put into the following algorithm. 


Algorithm 10.4.2 (Infeasible Primal-Dual Interior-Point Algorithm) Given 
0 <€,€p,€a <1, A € (0, 1). Initial: (x, y, Z) satisfying x, z > 0. This algorithm 
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solves the standard LP problem. 


1. Compute r, = AX — b, rg = ATY +Z—c. 

2. Stop if *1z/(1 + |bT¥|) < ©, rp < €p, Ta < €a (optimality achieved 
approximately). 

3. Solve system (ADA')Ay =Pp —AD(Z—rg)+uAZ~!e, where jz is determined 
by (10.67). 

4. Compute Az = rg — ATAy, Ax = D(Z— Az) — Zoe. 

5. Update y, z, x by (10.57), where a is determined by (10.58) and (10.59). 

6. Go to step 1. 


Convergence of the preceding algorithm is not guaranteed in theory, and there 
is no device to detect the infeasibility or unboundedness either. Nevertheless, this 
algorithm performs well and inspires an even better variant, introduced in the next 
section. 


10.4.3 Predictor-Corrector Primal—Dual Algorithm 


The Newton equation (10.61)~(10.63), used in the infeasible primal—dual method, 
is yielded from (10.49)~(10.51) by omitting the second-order term A X Az. Mehro- 
tra’s (1992) predictor—corrector primal—dual method introduces an additional cor- 
rection step, involving the second-order term, to improve the search direction (for 
simplicity, the prefix “infeasible” will be omitted). 

It is noted that the left-hand side of (10.49)~(10.51) is linear in Ax, Ay, Az, 
while only the right-hand side involves the second-order term AX Az. The system 
can therefore be regarded as an implicit one. We can determine a predictor solution, 
say (Ax’, Ay’, Az’), to approach its solution first, then substitute it to the right-hand 
side, and solve the resulting system for a corrector solution. 

Despite the Newton direction (10.64)~(10.66) is a clear choice for being 
predictor solution, Mehrotra uses the solution to the linear system below: 


AAx' = rp, 
ATAy’+ Az =ra, 
ZAx' +X Adz = Xz. 
The only difference between the preceding and Newton equation is that its right- 
hand side no longer involves term — je. It can be viewed as one derived from the 


complementarity condition Xz = 0 rather than Xs = we. So, its solution can be 
obtained easily from (10.64)~(10.66) by setting w = 0, ie., 


Ay! = (ADA') | (rp — ADG — ra)), (10.71) 
Az’ = rq — A'Ay’, (10.72) 
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Ax’! = D(Z— Az’). (10.73) 
Substituting Az’ and Ax’ to the right-hand side of (10.51), we solve 
AAx =TPrp, 
ATAy +Az=ra, 
ZAx + XAz =X Z +AX'Az' — pe, 


to obtain a corrector solution. Only the right-hand sides of the preceding system and 
(10.61)~(10.63) differ, hence the derivation of the corrector solution is similar to 
that of (10.64)~(10.66), i-e., 


Ay = Ay’ — (ADA')“!(AZ7!AX'Az’ — pAZ7!e), (10.74) 
Az=rq—A'Ay, (10.75) 
Ax = D(Z— Az) + Z~!AX’Az’ — pZ7e. (10.76) 


The predictor solution may also be used to determine barrier parameter jy. 
Mehrotra (1991) uses the following formula: 


we = (g/X"2)?(g"/n), (10.77) 


where 


g = (& — a, Ax')"(% — a, Az’), 
po, = 0.99995 minix;/ Ax | Ax > 0,7 1;.ac on), 


or!) = 0.99995 min{Z;/Az’, | Az’, > 0, j =1,...,n}. 


It is noted that g’ is a predicted dual gap, whereas x'Z is the current dual gap. Their 
ratio (less than 1) may be regarded as a predicted improvement at the iteration. So, 
a small jz is taken when the predicted improvement is significant, while a large yu is 
taken in the other case. This is appropriate because small improvement implies that 
the tendency toward the center should be enhanced. 

Mehrotra’s work draws the attention of the community immediately. To evaluate 
the predictor—corrector primal—dual method vs. the pure infeasible primal—dual 
method, Lustig et al. (1992) conducted further computational experiments, involv- 
ing 86 Netlib standard test problems. Their numerical results indicated that the 
former outperformed the latter with 85 out of the 86 problems, in terms of iteration 
counts; and most importantly, in terms of CPU time, the former defeated the latter 
with 71 out of the 86, exceeding 82%. Overall, it is well accepted that the predictor— 
corrector primal—dual method is superior to the infeasible primal—dual method. 
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In order to avoid solving ill-conditioned problems and instability when iterates 
are close to optimal solution, they modified the determination of the barrier 
parameter as follows: 

Use (10.77) only when x'Z > 1; otherwise, use 


pw = X'Z/o(n), (10.78) 


where ¢(7) is still defined by (10.70). 
Overall steps are put into the following algorithm. 


Algorithm 10.4.3 (Predictor—Corrector Primal-Dual Algorithm) Given 0 < 
€,€p,€a < 1, A € (0,1). Initial: (x, y, z) satisfying x,z > 0. This algorithm 
solves the standard LP problem. 


1. Computer, = Ax — b, rg = Aly+Z-c. 

2. Stop if TE/(1 + (bs) < rp < €p, Td < €d (Optimality achieved 
approximately). 

. Compute pz by (10.77) if x'Z> 1; else, by (10.78). 

. Compute Az’, Ax’ by (10.71)~(10.73). 

. Compute Ay, Az, Ax by (10.74)~(10.76). 

. Update y, z, x by (10.57), where a is determined by (10.58) and (10.59). 

. Go to step 1. 
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Now let us turn to how to obtain an initial point for the infeasible primal—dual 
method and the predictor—corrector primal—dual method, etc. 

It is attractive that the initial infeasible point seems to be quite free to choose. 
Since it can affect the merit of the algorithm considerably, the initial point should 
be nearly primal and dual feasible as well as close to the center. It turns out that there 
are still numerical difficulties if it is not properly centered even if being close to an 
optimal solution. Mehrotra (1992) suggests obtaining the initial point by solving a 
quadratic programming problem. Subsequently, Andersen, Gondzio, Meszaros, and 
Xu derive a variant, handling the following convex quadratic programming problem: 


min c!x + (n/2)x" x; 
s.t. Ax =b, 


where 7 > 0 is a weight factor. The Lagrange function of the preceding problem is 
L(x, w) = cx + (n/2)x'x — wl (Ax — bd). 
An explicit expression of the solution to VL (x, w) = 0 is then 
*® = (1/n)(Al(AA!)7! Ac — c). 


To be an initial solution, x’s components less than some positive number 6 are 
modified to 6 (e.g., 6 = 1). An initial dual solution (y, z) is determined similarly. 
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Although it works well in practice, the above heuristic approach is scaling 
dependent and does not certainly generate a well-centered initial point. For Algo- 
rithm 10.4.3, moreover, it is assumed that there exists an optimal solution since there 
is no reliable method to detect infeasibility/unboundedness, as is a serious drawback 
to practice. 


10.4.4 Homogeneous and Self-Dual Algorithm 


The so-called homogeneous and self-dual algorithm might be the most efficient 
interior-point method at present. It overcomes the two drawbacks of the predictor— 
corrector primal—dual algorithm, presented in the previous subsection, by a so-called 
homogeneous and self-dual model, which is based on skew-symmetric self-dual 
artificial model introduced by Ye et al. (1994), though it was subsequently found 
that Goldman and Tucker (1956) and Tucker (1956) studied such types of models 
even earlier. We will focus on main elements of it here in this subsection; for more 
details, the reader is referred to the relevant literature, such as Jansen et al. (1994), 
Xu and Ye (1995). 
We begin with the self-dual problem (5.9), written as 


min c'x — b'y, 
s.t. Ax —b = 0, x > 0, (10.79) 
—Aty +c-z=0, z>0, 


which is equivalent to the pair of primal and dual standard problems (5.1) and 
(5.2), in the sense that they share the same optimal solutions. We attempt to modify 
the preceding homogeneous and self-dual problem to a better “trivial” one, whose 
solution is infeasible to the original problem. 

Since the expected iterates will satisfy the nonnegative but not equality con- 
straints, we add 


b'y—clx —o =0, o > 0, 


to constraints and put the preceding problem into the following problem in variables 
(x, y, Z, 0, T): 


min 0O 
s.t Ax — bt = 0, 
—Aty +ct—z = 0, (10.80) 
bly —clx +o=0, 
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which is associated with (10.79) by o, t. According to Theorem 5.3.3, there exists a 
strictly complementary optimal solution, say (x*, y*, z*, t*, 0*), to (10.80), which 
can be shown to satisfy the following systems: 


Xz=0 to=0, 
10.81 
x+z>Ott+o>0. get) 


The following can be shown: 


(i) Case t* > 0. Then (x*/t* and (y*/t*, z*/t*) are a pair of strictly comple- 
mentary optimal solutions to (5.1) and (5.2). 

(ii) Case t* = 0. This implies o* > 0, and hence b' y* — c'x* > 0. Thus, at least 
one of b'y* > 0 and —c'x* > 0 holds: if b'y* > 0, the primal problem (5.1) 
is infeasible; if c'x* < 0, the dual problem (5.2) is infeasible, and if bt y* >0 
and c'x* < 0, then the pair problems are both infeasible. 


The requited strictly complementary optimal solution to (10.79) can be achieved 
in the limit via solving a series of “trivial” problems, modified from (10.80). To this 
end, introduce the feasibility and the average complementarity residual functions 
below: 


rp = bt — Ax, 

rq =Aly—ct+z, 
paar T 

rg =o x—b y—o, 


wm = (x'z+170)/(n+ 1). 


(10.82) 


Denote by the symbol “e” the quantity at the current iterate (y,x > 0,T > 
0,z > 0,0 > OQ). Problem (10.80) is modified by putting the complementarity 
residual functions and logarithmic barrier terms in its objective and laying current 
residuals in the place of the right-hand side of the constraints. As in the primal— 
dual method, the logarithmic barrier terms are for centering and keeping iterate’s 
positiveness. The resulting problem is then 


n 
min z'x +or— Ap Sdn; + Inzj) — Ap (Int +Ino), 


j=l 
s.t. Ax — bt = —hFp, (10.83) 
—Aly +ct—z =—Arg, 
bly —c™x +o =—Ar,, 


where A ¢€ [0, 1] is called the path parameter. It is noted that the current iterate is 
feasible to (10.83). 
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The first optimality conditions for (10.83) are 


Ax — bt =—Arp, 
—Aly +ct—-—z =—-Arq, 
b'y —clx +o=—Arg, (10.84) 
XZ = he, 
to = Ap, 


X,Z,0,T > 0. 


Similar to the primal—dual method, the search direction is determined by 
applying Newton’s method to (10.84). Denoting by y ¢€ [0, 1] a predetermined 
reduction rate of the path parameter, the resulting Newton equation for the search 
direction (Ax, Ay, Az, Ao, At) can be written as 


A —b Ay (l—y)rp 
—AtT c -l Ax (l—y)ra 
bY =e! -1 At |=| (—y)re 
Z, X Az y pe — XZ 
o T Ao yu—to 
Setting y = 1 gives a pure centering direction, and setting y = O gives an 


affine direction. Such a system is solved in each iteration, constituting the major 
computational task in the method. 

Once the Newton direction has been computed, a stepsize is determined by using 
the same formulas as in the primal—dual method so that the new iterates are strictly 
positive. It is self-evident that the termination criterion should be set properly. 


10.5 Notes on Interior-Point Algorithm 


It is known so far that the earliest interior-point algorithm is proposed by famous 
mathematician von Neumann in a talk with G.B. Dantzig in 1948 (Dantzig and 
Thapa 2003). It is designed, without convergence proof, to generate a feasible 
solution to a special LP problem, which turns out to be just the Karmarkar standard 
problem. An elegant proof was given by Dantzig in a letter to J. von Neumann. 

Originally, the logarithmic barrier function was used by Frisch (1955) to design 
interior-point methods for solving nonlinear programming problems. This nonlinear 
tool was applied to LP only after the publication of the Karmarkar algorithm (1984), 
inspiring a boom of interior-point methods. It was utilized by the path-following 
method to create the search direction that is a combination of a descent direction 
and a centering direction. The other interior-point methods presented in this chapter 
can also be derived alternatively in such a way (Gill et al. 1985-1986; Shanno and 
Bagchi 1990). 
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To explain, we begin with the nonlinear programming problem (10.43). The 
gradient and Hessian matrix of its objective function at current interior point, say 
xX, are, respectively, 


Vf(x)=c—pX le, V* f(x) = ux. 


It is noted that V7 f (x) is positive definite. Setting x = x—A.x, the problem becomes 
one with respect to Ax, and the latter can be approximated by the following strict 
convex quadratic programming problem: 


min f (x) — Vf (x)TAx + (1/2)AxTV? f (x) Ax 
s.t. AAx = 0. 


Thus, what needs to do is to find the stationary point of the following Lagrange 
function: 


L(Ax, y) = —V f()' Ax + (1/2)Ax'V? f (#)Ax — yTAAx, 


where y is the Lagrange multiplier vector, by solving system 


px AeA yao pr ‘ea, (10.85) 
AAx = 0. (10.86) 

It holds by (10.85) that 
Ax = (1/p)(X7Aly + X2c — Xe), (10.87) 


substituting which to (10.86) gives 
y = —(AX7A")“! AX (Xe — pe). 
Then, substituting the preceding to (10.87) leads to 
Ax = (1/u)X P(Xc — pe), (10.88) 


where P is the projection matrix, defined by (10.26). —Ax is the Newton direction 
of problem (10.43) at x. It is seen by comparing with (10.27) that the vector is just 
a positive combination of X Pe and the search direction —X Ax’ = —X PXc, used 
by the primal affine algorithm. 

Assume that the feasible region of the original problem is nonempty. Consider 
problem 


n 
min ) Inx;, 
j=l 


s.t. Ax =b, 
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to which there is a unique optimal solution that can be regarded as the center of the 
feasible region. It is not difficult to show that X Pe is just the Newton direction of the 
preceding problem at x. Therefore, the search direction, defined by (10.88), reflects 
the common effect of a “descent force” and a “centering force.” The size of the 
barrier parameter determines the proportion of the two tendencies. Algorithms with 
polynomial time complexity can be obtained by properly selecting the parameters 
and stepsize (Gonzaga 1987). On the other hand, the search direction tends to the 
one, used by the primal affine algorithm, as jz tends to zero. The latter is not of 
polynomial-time complexity. 

With the same logarithmic barrier function, it is possible to derive the search 
direction, used by the Karmarkar algorithm. The resulting algorithm is equivalent to 
the Karmarkar algorithm if a proper barrier parameter and stepsize are taken. 

As for the dual problem, construct the following nonlinear programming prob- 
lem: 


n 
max g(y) =bTy +) In(cj — 47), 
j=l 
where a; denotes j-indexed column of A. Introduce 
Z=c—Aly. 


The gradient and Hessian matrix of the objective function at the current point y are, 
respectively, 


Vg(v) =b—pwAZ'e, V7 g(3) =—wAZ~? Al. 
The Newton direction is then 
Ay = —(AZ~*A")"!(b/w + AZ~'e), 
which is a positive combination of —AZ~'e and the search direction, used by the 


dual affine algorithm, and the former is Newton direction, somehow pointing to the 
neighborhood of the center of the dual feasible region of the following problem: 


n 
max > In(e; _ aj). 
j=l 


Another nonlinear tool applicable to interior-point method is ordinary differential 
equations (ODE). Let us bring up the affine algorithm. Its search direction 


=N(Po RA (AR AY AM Ke 
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actually determines a vector field for the feasible region. Taking it as right-hand 
side, we construct the following autonomous (ODE) system: 


ao = —X(I — XAT(AX2A1)“!AX)Xc, x(0) =, (10.89) 


where x is an initial interior point. Adler et al. (1989) realized the dual affine 
method and reported remarkable computational results. They defined a trajectory 
by ordinary differential equations (ODEs) and used the power series numerical 
integration of the first and second order to obtain an approximate optimal solution. 

As for the ODE method, it is worth making further remarks not just because it 
is successful in realizing the dual affine method but also because it is of general 
significance. In fact, for any linear or smooth nonlinear programming methods, 
the ODE system can be introduced similarly. Under mild conditions, there exists 
a trajectory to such a system, whose limit point is an optimal solution (Pan 1992a; 
1992b). More precisely, let x(t),0 < t < y, be the right-hand half of (10.89)’s 
trajectory, starting from x, such that 


x(t) > x* as ty, 


where x* is the optimal solution. It is not difficult to solve it approximately by some 
numerical integration method. 

According to the author, the earliest: ODE method was proposed by Arrow 
and Hurwicz (1956) for solving equality constraint nonlinear programming prob- 
lems. There were many related papers published subsequently (e.g., Fiacco and 
Mccormick 1968; Abadie and Corpentier 1969; Evtushenko 1974; Botsaris 1974; 
Tanabe 1977). A common belief seems that the computational cost related to the 
ODE is quite high, compared with standard methods, such as SQP, even if Brown 
and Bartholomew-Biggs (1987) reported very favorable numerical results with the 
former. The author himself has also experienced the success of the ODE method 
(Pan 1982, 1992b) and is surprised why it has been ignored by the academic 
community. 

After all, the normal line search is only of the first order in certain sense; taking 
small stepsize in Algorithm 10.2.1 amounts to executing numerical integration with 
the first-order power series method, or Euler method. In particular, realizing curve 
search via the second-order power series integration should apply to nonlinear 
programming problems. Based on this idea, the algorithm using higher order search 
scheme outperformed both DFP and BFGS methods significantly in computational 
experiments (Shi and Pan 2011). 

Finally, we conclude this chapter with the following remarks. 

The interior-point method is very successful. Some scholars even believe that 
it is better than the simplex method for solving large-scale sparse LP problems. 
The homogeneous and self-dual method has already been implemented in some 
commercial codes, such as CPLEX, and is deemed as one of the most powerful 
methods. Such types of methods can be well parallelized, compared with the 
simplex method. 
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Despite all these, the interior-point method has been seriously limited in applica- 
tions due to its inherent weakness. It only provides an approximate optimal solution 
and hence needs an additional purification procedure in applications requiring an 
optimal vertex solution. Moreover, it cannot be warmly started, compared with the 
simplex method that achieves optimality quickly if starting from some basis reached 
in a previous solution process. Consequently, the interior-point method does not 
apply to integer or mixed-integer LP programming, since solving such problems 
yields a large number of correlated LP problems, which can only be left for the 
simplex method. At present, therefore, the dominant position of the simplex method 
is difficult to shake. 

Now a race between the simplex method and the interior-point method seems 
to continue. No single method applies to all LP problems. As a result, there are 
usually multiple options of methods included in professional codes. To take the 
advantage of both types, some scholars first apply the interior-point method, and 
then the simplex method (Bixby et al. 1992), although it seems too mechanized. 
In Part II of this book, we will introduce recently developed pivotal interior-point 
methods in a natural manner. 
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Chapter 11 
Integer Linear Programming (ILP) crests 


The feasible region of the LP problem is continuous since each variable is restricted 
to a continuous interval. If variables (or a part of variables) are further restricted 
to integer values, it becomes an integer LP (ILP) (or mixed-integer (MILP)) 
problem that differs from LP problem essentially. This chapter highlights topics, 
such as graphic approach and basic idea, cutting-plane method, branch-and-bound 
method, controlled-cutting method, controlled-branch method, including depth- 
oriented strategy and breadth-oriented strategy. The final section introduces an 
actualization of the controlled-branch method with the reduced simplex framework, 
which is dealt with in detail in Part IL. 

A vector/point is called integer if all of its components are integer. The definition 
of the feasible set of the ILP problem is obvious. 

In the next section, a graphic approach to ILP is illustrated with a 2-dimensional 
instance to introduce basic ideas behind ILP solvers. The following two sections 
are devoted to cutting-plane and branch-and-bound methods, respectively. Then, we 
develop new ILP methods, named controlled-cutting and controlled-branch in the 
other two sections. We present a realization of the controlled-branch method in the 
final section. 

In this chapter, we denote the maximum integer not greater than e by 


Le 


and the minimum integer not less than e by 


Vel 
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11.1 Graphic Approach 


The convex hull of the feasible set is termed integer hull of the ILP problem. It can 
be shown that vertices of the integer hull are integers. Therefore, the task is boiled 
down to solving the corresponding relaxation LP problem over the integer hull. 

This can be illustrated by solving the following 2-dimensional instance graphi- 
cally: 


min f = —2x1 = 5x2 
(LP) s.t. 2x1, + 3x2 < 12 
xytx2< 5 (11.1) 
x2 < 3 
integer x1, x2 > 0. 


The optimal solution to the LP relaxation of the preceding is xj = 1.5.x2 = 3, with 
optimal value f = —18 (see Sect. 2.6). 

As sketched in Fig. 11.1, the feasible region of the LP relaxation is the area 
enclosed by polygon OABCD. The dots constitute the feasible set of (11.1), and the 
area enclosed by polygon OAEFD is the integer hull. We solve the problem over the 
integer hull by the following LP graphic approach. 

In the figure, the isoline —2x; —5x2 = 0 of the objection function passes through 
the origin. The objective value reduces from 0 as the line shifts parallel to the upper- 
right direction. It is seen that the farthest possible position, with respect to the area 
of OAEFD, is line —2x; —5x2 = —17, passing through integer vertex E. Therefore, 
the optimal solution to ILP problem (11.1) is xf = 1,x3 = 3, f* =—17. 

Although the graphic approach is only of conceptual sense, it would give some 
idea about the creation of some ILP solvers. 


Definition 11.1.1 The LP relaxation of an ILP problem is the LP problem resulting 
from dropping its integer restriction on variables. 


ILP problem and its LP relaxation form a 1-to-1 correspondence. For example, 
the LP relaxation of the ILP problem (11.1) is over the feasible region, the area 
surrounded by OABCD. 

The validity of the following is obvious. 


Proposition 11.1.1 Jf the optimal solution to the LP relaxation is an integer 
solution, it is optimal for the ILP problem. 


The preceding forms a common ground of the existing ILP solvers, like the 
cutting-plane and branch-and-bound methods. At first, it solves the LP relaxation of 
the ILP problem. It is done if the obtained is an integer optimal solution. Otherwise, 
it deals with the LP relaxation according to its idea by solving a series of LP 
relaxations, which are so rich in practice, as to form a major application area of 
LP at present. 
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221 + 542 = 0 


Fig. 11.1 Graphic solution to ILP problem 11.1 


It turns out that only a finite number of LP relaxations need to be solved. 
However, the finiteness is of theoretical value only. In practice, any of the solvers 
alone is far from satisfactory. Therefore, some combination of them is now widely 
used in the ILP area. We will not go into details though. 

In addition, it is observed that if one cuts off some part of the feasible region 
OABCD, which does not include the feasible set of the ILP problem, then the rest 
may be closer to the integer hull OAEFD; in the algebraic words, adding some 
appropriate inequalities to the constraints would lead to a better LP relaxation. This 
is another common point of the existing ILP solvers. 

The valid cut used is defined as follows. 


Definition 11.1.2 A valid cut of an ILP problem is an inequality, satisfied by all 
feasible solutions to it but violated by the optimal solution to its LP relaxation. 


On the other hand, one can divide OABCD into several parts and seek integer 
optimal solutions over each of them. This means solving the corresponding so-called 
LP subrelaxations and comparing their end products. This is the basic idea behind 
the branch-and-bound method. 
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The resulting parts are called branches of the feasible region of the LP relaxation. 
The feasible region of the ILP problem is the largest (trivial) branch. Without 
confusion, the corresponding LP subrelaxations as well as ILP subproblems are also 
said to be branches. 

In this aspect, the basic tool is the so-called valid pair of integer cuts, which 
divides OABCD into two branches. The pair cuts off a narrow area with a thickness 
of one unit without affecting the feasible set of the ILP problem. 


11.1.1 Basic Idea Behind New ILP Solvers 


Assume that the coefficients of the objective function are all integers and hence so 
are the objective values associated with the feasible (integer) solutions. 

From the Fig. 11.1, it is observed that the wanted optimal integer solution lies on 
the isoline, —2x; —5x2 = —17, of the objective function of the LP relaxation, which 
is a line close to its lowest isoline passing through vertex B, i.e., —2x;-—5x2 = —18. 


Definition 11.1.3 Let f~ be an integer lower bound on the optimal value of the 
ILP problem. The inequality 


clx > f- (11.2) 


is an objective cut. 
Then, the validity of the following claim is obvious. 


Lemma 11.1.1 Let f~ be an integer lower bound on the optimal value of the ILP 
problem. If an optimal solution to an LP subrelaxation is an integer solution that 
attains the objective value equaling to f~, it is optimal for the ILP problem. 


The preceding forms the footstone of the new ILP solvers. 
But, how to obtain a tight integer lower bound on the optimal value? If the 


optimal value, f, of the LP relaxation is available, we take the following: 
fi ="f. (11.3) 


Thus, the following reveals the basic idea behind the new solvers. 


Proposition 11.1.2 (Heuristic Characteristic of the Optimal Value) The optimal 
value of the ILP problem is towards f and tends to reach f~— =" f. 


It is clear that if the range of the objective value of the relaxation is no more than 
one unit, the optimal solution of the LP relaxation modified by adding the objective 
cut (11.2) must attain the objective value being equal to f—. If further, it is an integer 
solution, it is optimal for the ILP problem, according to Lemma 11.1.1. Otherwise, 
we proceed by adding some cuts and solving corresponding LP subrelaxations. 
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11.2 Cutting-Plane Method 


Many authors contributed to the cutting-plane method. The method presented in 
this section essentially follows Gomory (1958). It solves a series of LP relaxation 
problems associated with ILP subproblems, generated from adding valid cut 
successively. It terminates when an optimal solution to some LP relaxation is an 
integer, which is also optimal to the ILP problem. Thus, the idea of the method is 
quite simple. 

The big problem is how to generate good cuts to speed up the solution process. 
Extensive research has been done in this aspect. In this section, only the so-called 
fractional cut is utilized for tasting. 

Let B = {ji,..., jm} and N = A\B be optimal basis and nonbasis of the 
LP relaxation, respectively. Assume that the associated optimal basic solution is 
noninteger. Specifically, assume that the equation corresponding to the ith row of 
the optimal simplex tableau, i.e., 


xj, + Djen Gj, jx; = bj (11.4) 


is of noninteger right-hand side b;. 
Introduce the following quantities: 


0 < aj, j = Gj, ; — La; ;4, 0 < Bi =b; —Lbjs <1 
by which, (11.4) can be written 
Xj, + Ujen (LGi, jit aj, j)xj = Lb a+ Bi > 0. 


Converting the preceding by moving the terms with integer coefficients to the left- 
hand side and those with fractions to the right-hand side gives 


Xj, + Ljen (Lj, j4)X; —cbis = Bj — Xjen Qj, jXj- 


For any integer feasible solution x, the left-hand side of the preceding is an integer, 
and hence so is the right-hand side. In addition, since 


aij = 0, xj > 0, VI EN, 


the right-hand is no more than 0 < £; < 1. Therefore, both sides must be an integer 
less than and equal to 0, i.e., 


integer Bi - LjeN Oj, jXj S 0. (11.5) 


It is then asserted that the associated optimal basic solution does not satisfy (11.5). 
In fact, if it is substituted to (11.5), the left-hand side becomes f; > 0 because all its 
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nonbasic components are zero. Therefore, since it includes the whole feasible set, 
(11.5) is a valid cut. 

The ILP problem is modified by adding such a cut, and the associated LP 
relaxation is solved then. If the yielded solution is still noninteger, a new valid cut 
is determined, and added to the constraints, and so on until a resulting solution is an 
integer, which is an optimal solution to the original ILP problem. 

Optimal solution to LP relaxation would have multiple noninteger components, 
and hence there are multiple choices of valid cut. In this case, it seems to be 
favorable to form a cut cutting deepest into the feasible region of the LP relaxation. 

To this end, let J be the set of row index corresponding to noninteger right-hand 
side. For anyi € 7, 


2 

Bj / x JEN Ori 
is the Euclidean distance from the associated solution to the LP relaxation to the 
boundary of the area defined by (11.5) (Sect. 2.1). Unfortunately, the associated 
computations are cumbersome. Instead, we may use residuals approximately by 
determining row index i’ such that 

i’ € arg max f;. 

ie] 


Example 11.2.1 Solve the following ILP problem by the cutting-plane method: 


min f = —48x) — 40x2 


s.t. 8x] + x3 — 3x4 = (11.6) 
—2x1 + 2x2 + 2x4 = 12 
integersxj > 0, j=1,...,4. 


Answer Call simplex Algorithm 3.1.1, yielding 


Xp x2 x3 x4 RHS 
1 1/8 —3/8 | 1/4 
1 1/8 5/8 | 25/4 

11/8 7/8 | 131/4 


Arbitrarily, take the first row with noninteger right-hand side. The corresponding 
equation is 


x1 + 1/8x3 — 3/8x4 = 1/4, 
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or equivalently 
xy + 1/8x3 + (—1+5/8)x4 = 1/4. 
The preceding can be written 
x1 — x4 = 1/4— 1/8x3 — 5/8x4. 
From which the valid cut follows, i.e., 
1/4 — 1/8x3 —5/8x4 < 0. 


Insert the row corresponding to —1/8x3 — 5/8x4 + x5 = —1/4 to the preceding 
tableau as the second bottom row, giving 


XxX] x2 X3 X4 
1 1/8 3/8 
1 1/8 5/8 
21. =578 
11/8 7/8 


which is a dual feasible simplex tableau. Call the dual simplex Algorithm 5.4.1, 
yielding 


Arbitrarily, take the third row with noninteger right-hand side. The corresponding 
equation is 


1/5x3 + x4 — 8/5x5 = 2/5, 
or equivalently 
1/5x3 + x4 + (—2 + 2/5)x5 = 2/5. 
The preceding can be written 


xq — 2x5 = 2/5 — 1/5x3 — 2/5xs. 
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From which the valid cut follows, i.e., 
2/5 — 1/5x3 — 2/5x5 < 0 


Insert the row corresponding to —1/5x3 — 2/5x5 + x6 = —2/5 to the preceding 
tableau as the second bottom row, giving which is dual feasible. Call Algo- 


XxX] x2 X3 x4 X5 X6 RHS 
1 1/5 —3/5 2/5 
1 0 1 6 
1/5 1 —8/5 2/5 
-1/5 —2/5 1 —2/5 
6/5 7/5 162/5 


rithm 5.4.1, yielding 


Xp X92 x3 X4 X5 X6 RHS 
1 1/2 —3/2 | 1 
1 -1/2 5/2 | 5 

1 1 —4 2 

1/2 1 —5/2 | 1 

1/2 7/2 31 


which give an integer solution, which is optimal to the original ILP problem, i.e., 


cae aS = 5, f* =31. 


11.3. Branch-and-Bound Method 


Let P and x be the feasible region and the optimal solution to the LP relaxation 
problem of the ILP problem. 

If x is noninteger, we need to divide the LP relaxation into a pair of LP 
subrelaxations. To do so, introduce the following. 


Definition 11.3.1 Integer cut is inequality corresponding to hyperplane passing 
through integer point in an axis, and being perpendicular to the axis. 


Let x; be fractional component of x. Then cuts x; < Lx; and x; = x7! are 
said valid pair of integer cuts, and the variable x; is cutting variable. Adding the 
two cuts divides P into a pair of LP subrelaxations with 


PO{x|xj<cxjs} and = PA {x | xj > "E/J} 
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as their feasible regions, respectively, including the feasible set of the ILP problem, 
but cutting out the area 


Xx; ; ry. 
LX 74 < Xj < Xj ‘ 


LP subrelaxation problem or the associated ILP subproblem is also called branch. 

The branch-and-bound method was developed by Land and Doig (1960). It 
solves a series of branches, yielding from adding integer cuts to seek an optimal 
solution to the ILP problem, as described as follows. 

Set ft = +00 as the initial upper bound on optimal value f* to the ILP problem. 

Each time, one usually faces multiple branches. One of them is solved to find 
its optimal solution, and the others are left pending, including those deemed as 
fathomed. The former and related integer cut are said active. 

The fathomed means that there is no need for the branch to be divided further, as 
is in one of the following cases: 


(i) The branch is infeasible. 
(ii) The solution to the branch is an integer. 
It is clearly feasible to the ILP problem. Update f+ = f if the correspond- 
ing objective value f is strictly less than the current upper bound. 
(iii) The solution is noninteger, but it holds that" f1 > ft. 


The ILP problem is solved when all branches are fathomed. 

The branch-and-bound method generates a series of branches that form the so- 
called enumeration tree. There are breadth-oriented and depth-oriented strategies 
to develop enumerate tree. Once a branch is found not fathomed, the former solves 
the nearest pending branch (in the same level), and the latter solves one of the 
branches at a deeper level while leaving the other branch pending, until reaching 
a fathomed branch. Then, the pending branches are handled one by one, from the 
nearest (deepest) to the top level, until all branches are examined. 

We take the example (11.1) to show details of the branch-and-bound method with 
depth-oriented strategy. 

The solution to the LP relaxation of (11.1) is x} = 3/2, x2 = 3, 7 = —18. Since 
component x; = 15 is noninteger, we add the valid pair of cuts, x1 < | and x, > 2 
to generate the following pair of ILP subprograms: 


min f = —22 — 522 min f =—22, —5x 
s.t. 22, +32%2 < 12 s.t 24, +322 < 12 
(ILP1) Mt+ta < 5 (ILP2) mamta < 5 
wm < 3 wm < 3 
%< 1 x > 2 
integer 71, % > O integer 71, t2 > O 
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Fig. 11.2 Feasible sets for the two ILP subprograms 


Figure 11.2 depicts the corresponding two branches with L1 and L2 as their 
feasible regions, respectively. It is noted that the solution B to the LP relaxation is 
excluded from L1 M L2. 

Therefore, solving the ILP problem (11.1) is boiled down to solving subprograms 
(ILP1) and (ILP2), separately. We arbitrarily select, and solve the branch (ILP2), and 
leave (ILP1) pending, and so on. The solution process is illustrated via the simplex 
framework, as is follows. 


Example 11.3.1 Solve ILP problem (11.1) by the branch-and-bound method. 


Answer Set initial upper bound on the optimal value by f* = +00. Convert the 
problem to the standard form by introducing slack variable x3, x4, x5. The initial 
simplex tableau of the associated LP relaxation is then 


x] X2 x3 x4 \X5 | RHS 
2 3 1 12 
1 1 1 5 

1 1 3 
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(1) Call simplex Algorithm 3.1.1, yielding 


xX] x2 X3 X4 
1 1/2 
-1/2 1 
| 
1 
(2) Selecting integer cut x; > 2, add the row, corresponding to —x; + x6 = —2, to 


the preceding as the second bottom row. 


x] x2 x3 X4 x5 
1 1/2 =3/2 
=172 ft ‘18 
1 1 
—1 
1 2 


Convert the preceding to the simplex tableau, and call dual simplex Algo- 
rithm 5.4.1, yielding 


xX] 9) X3 X4 X5 
1 
ai - 4 
1 1/3 
=i 1 
5/3 


(3) Selecting integer cut x2 < 2, add the row corresponding to x2 + x7 = 2 to the 
preceding as the second bottom row. 


xX] x2 X3 X4 X5 X6 
1 il 
-13 14 1/3 
1 1/3 2/3 
13 i 42/3 


5/3 4/3 
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Convert the preceding to a simplex tableau, and call dual simplex Algo- 
rithm 5.4.1, yielding 


XxX] x2 X3 X4 X5 X6 
1 1/2 
-1/2 1 
1 
1 
1/2 1 


which gives a feasible (integer) solution to the ILP problem (fathomed branch). 

Updated the upper bound by f+ = —16. 

(4) Selecting integer cut x2 > 3, add the row corresponding to —x2 + x7 = —3 to 
the end tableau of step (2) as the second bottom row. 


Pal x2 X3 X4  X5 X6 x7 
1 -1 
-1/3 1 1/3 
1 1/3 2/3 
1/3 i ape 
-1 1 
5/3 4/3 


Convert the preceding to a simplex tableau, and call dual simplex Algo- 
rithm 5.4.1, yielding 


x1 x2 X3 X4 x5 X6 XT 
1 =! 
=1j73 1 1/3 
1 17 2/3 
—1/3 1 2/3 
1/3 o/8 4 
5/3 4/3 


which indicates that there is no feasible solution to the subprogram (fathomed 


branch). 
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(5) Selecting integer cut x; < 1, add the row corresponding to x; + x6 = | to the 
end tableau of step (1) as the second bottom row. 


Xp X92 x3 x4 x5 x6 | RHS 
1 1/2 —3/2 3/2 
—-1/2 1 1/2 1/2 

1 1 3 

1 1 1 
1 2, 18 


Convert the preceding to a simplex tableau, and call dual simplex Algo- 
rithm 5.4.1, yielding 


which gives a feasible (integer) solution to the original ILP problem. A comparison 
with step (3) indicates that it is optimal, i.e., 


x* = (1,3, 1, 1,0,0)", f* =—17. 


Figure 11.3 gives the corresponding enumerate tree, showing branches (1) though 
(5), handled in the solution process. 

It is clear that enumerate tree is not unique. The branch-and-bound method 
would still generate an enumeration tree with an unacceptable large number of 
branches. It could perform unsatisfactorily or even fail with some relatively small 
ILP problems. 

The issue involves a double decision: how to select a cutting variable, and how 
to select one from the valid pair of integer cuts. Different decisions lead to different 
computational costs. The classical branch-and-bound method makes a decision 
arbitrarily, as was shown in the instance. Nevertheless, the effect of a decision cannot 
be ignored, and it is wise to do according to some appropriate criteria rather than 
blindly. 

Let J be the index set of noninteger components of the current solution x. 
To select a cutting variable, the following rule depends on the contribution to the 
optimal value. 
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Fig. 11.3, An enumerate tree (1) 
to ILP problem (11.1) 


f=-173 ft =-17 
tw <2 w>3 
(3) (4) 
f= 3 Ho = 2 infeasible branch 


Rule 11.3.1 Determine cutting variable xj: such that 
j’ € arg max |cjx;|. (11.7) 
jet 7 * 
Another rule depends on objective coefficients only. 
Rule 11.3.2 Determine cutting variable x j such that 


j’ € argmax |c;|. (11.8) 
Jed : 


An alternative consideration is to cut out deepest into the feasible region of the 
LP relaxation, as is follows. 
For any j € J, 


aj = Xj = LX js 


is the distance from x to the boundary of {x | x; < Lx;4}, and 


is the distance from x to the boundary of x; > x; ' (Definition 2.1.2). Determine 
J’ such that 


6; = max{max a@;, max Bj}, 
jeJ jet 


and take x; as branching variable. Then, if 6;,5 = aj, choose cut xj < Lxj4; 
otherwise, choose cut x; > "xj; '. 
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Fig. 11.4 Another enumerate (1) 
tree to ILP problem (11.1) 
T=15 %=3 
f=-18 
Uy >2 L1 <1 
(2) (3) 
“= ry =3 %=2 %=8/3 
ft =-17 f=-17 


Based on the most-obtuse-angle heuristic (Proposition 2.7.1), we recommend the 
following as the tentative criteria. 


Rule 11.3.3 From the valid pair of variable cuts, choose one as active cut, whose 
normal direction (pointing to the interior of the half-space) forms an obtuse angle 
with the negative objective gradient. 


Specifically, assume that index j’ is determined. Then, if cj, < 0, select xj; < 
LX jr 45 and if cj, > 0, select xj: = x; '. In case of a tie, turn to the deepest cutting 
or the maximum |x ;’|. 

According to the preceding rule, branch (ILP1), rather than (ILP2), should be 
chosen to be active. 

Solution x; = 1, x2. = 3, fi = —17 to branch (ILP1) is integer, and hence it 
is fathomed with setting upper bound f+ = —17. Besides, since the solution to 
branch (ILP2) is x; = 2, x2 = 23, f= -174, and it holds that" f1 > f+, (LP2) 
is also fathomed. Therefore, it can be asserted that x; = 1, x2 = 3, f = —I7is 
optimal to the ILP problem. 

Consequently, Example (11.1) is solved by 3 calls of LP solver, while solved by 
5 calls previously, as is a considerable difference. 

Figure 11.4 gives the corresponding enumerate tree with less branches. 


11.4 Controlled-Cutting Method 


From now on, it is assumed that coefficients of the objective function are integers. 
This method utilizes the objective cutting in addition to the normal valid cutting, 
compared with the cutting-plane method that uses the latter only. 
Let f be the objective value of the obtained optimal solution to the LP relaxation 
of the ILP problem. If f is noninteger, we take the integer lower bound f~ on the 
optimal value by (11.3) and add the associated objective cut (11.2), ie., 


clx > f- 
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to the constraints. Then, solve the resulting LP relaxation. It is obvious that the 
objective value of its optimal solution must be equal to f . If the solution is integer, 
it is also optimal for the original ILP problem. 

If the solution is noninteger, add a normal valid cut, and solve the resulting LP 
relaxation, and so on, until achieving an optimal integer solution attaining an integer 
lower bound on the optimal value of the ILP problem. 

The objective cutting and valid cutting can be used alternately or flexibly. 


Example 11.4.1 Solve the following ILP problem by the controlled-cutting method: 


min f = —5x,;+ x2 

s.t. —7xy +42 +.%3 =4 
2x1 + 5x2 + x4 =7 
integerx; > 0, j =1,...,4. 


Answer (1) Call simplex Algorithm 3.1.1, yielding 


x1 x2 X3 x4 | RHS 
37/2 1 7/2 | 57/2 
1 5/2 1/2 | 7/2 
27/2 5/2 | 35/2 
(2) a = —35/2, f~— = —17. Add the row corresponding to objective cut 5x; — 


x2 + x5 = 17 to the preceding as its second bottom row: 


XxX] x2 X3 X4 X5 ‘| RHS~ 
37/2, 17/2 57/2 
1 5/2 ie 7/2 
5 1 1 17 
27/2 5/2 35/2 


After converting the preceding to a simplex tableau, call dual simplex Algo- 
rithm 5.4.1, yielding 


xX} xX.  X3 x4 x5 RHS : 
1 2/27) = 37/27 | 751/27 
1 1/27 5/27 92/27 
1 5/27  —2/27 1/27 


1 17 
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(3) Add the row associated with the fractional cut —2/27x4 — 10/27x5 + x6 = 
—22/27, formed from the first row of the preceding, as the second bottom row, 
leading to 


Xp x2 X3 x4 x5 X6 RHS 
1 2/27 37/27 751/27 

1 1/27 5/27 92/27 
1 5/27 —2/27 1/27 
—2/27 — —10/27 1 —22/27 

1 17 


bal x2 Xa X4 Xs X6 RHS 
1 1 7/2 25 

1 1/2 s} 
-1 1 —5/2 2 
5 1 -1 1 
1 5/2 15 


which gives an integer solution, which is also an optimal solution to the ILP 
problem, i.e., 


e=6,0,25,17, frS=i5, 


11.5 Controlled-Branch Method 


As was mentioned, the branch-and-bound method is far from satisfactory, although 
current commercial/professional ILP solvers are all rooted in it. We introduce the 
so-called controlled-branch method in this section. 

Its distinguishing feature is the use of objective cutting in addition to a valid pair 
of integer cutting. 

As usual, the method solves the LP relaxation of the ILP problem at first. If 
the optimal solution is integer, it is optimal for the ILP problem. Assume that it is 
noninteger with objective value f. 

The method sets the integer upper bound on the optimal value of the ILP problem 
by 
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At the same time, it takes the integer lower bound on the optimal value by 


It adds the objective cut 


cl x > f- 


to the constraints, and solve the resulting LP relaxation. It is clear that the objective 
value of its optimal solution is equal to f~ (assuming the range of its feasible values 
is no more than one unit). If the solution itself is integer, it is also optimal for 
the original ILP problem. Otherwise, we determine a cutting variable and add the 
associated valid pair of integer cuts to get two branches. 

Then, there are the two strategies to proceed, as follows. 


11.5.1 Depth-Oriented Strategy 


Facing a pair of branches, this strategy solves one of them and leaves the other 
pending. Then, it handles the active former just as before by further objective 
cutting; and so on. A series of branches are solved until reaching a so-called 
fathomed branch, as is in one of the following cases: 


(i) Current branch is infeasible. 
(ii) The solution to the current branch is an integer. 
The solution is feasible to the ILP problem. If the corresponding objective 
value f is strictly less than f+, update ft = f as well as the associated 
objective cut. 
(iii) The solution is noninteger and it holds that" f1 > f*. 


If the latter condition does not hold, update f~ =~ f7,! and continue. 

The ILP problem is solved when all branches are fathomed. 

Using the simplex framework, the following example demonstrates the realiza- 
tion of the controlled-branch method with the depth-oriented strategy. 


Example 11.5.1 Solve the following ILP problem by the controlled-branch method 
with the depth-oriented strategy: 


min f = 6x, + x2 + 3x3 
s.t. 2x1 + 5x3 
4x1, + 3x2 + x3 
integer x; > 0, j = 1,2,3. 


<9 
<5 


! f— is only an integer lower bound on the optimal value of the current ILP subproblem, not the 
original ILP problem. 
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Answer Introduce slack variable x4, x5. The optimal simplex tableau of its LP 
relaxation is 


(1) 
x1 X2 1X3 X4 x5 RHS 
[2/5 1 -1/5 9/5 
6/5 1 1/15 —1/3 16/15 
18/5 8/15 1/3 —97/15 


(2) Add objective cut —6x; — x2 — 3x3 + x6 = —7 to the preceding as its second 
bottom row: 


x] x2 X3 x4 x5 
2/5 1 -1/5 

6/5 1 1/15 —1/3 
—6 —1) 3 

18/5 8/15 1/3 


al x2 x3 x4 x5 x6 
[- (S90t anor, 9 
1 -1/9 —4/9 1/3 

1 4/27 5/54. 5/18 


(3) Arbitrarily selecting cutting variable x3, add the row corresponding to integer 
cut x3 + x7 = | to the preceding as the third bottom row: ] 


XxX] x2 X3 X4 x5 X6 
1 -7/27 -1/27 1/9 
1 -1/9  —4/9 1/3 

1 0 


1 4/27 5/54. —5/18 
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the optimal simplex tableau to which is The integer solution is obtained, and the 


xX] x2 X3 X4 X5 X6 
1 
=3 = 1 
= =3 1 
1 -1/2 
1 3 


branch is fathomed. Set upper bound ft = 15 


(4) Examine the pending branch. Add the row corresponding to integer cut x3 > 2 
to the end tableau of (2) as its third bottom row: 


Pal x2 x3 X4 x5 X6 
1 —7/27 = —1/27 1/9 
1 -1/9 —4/9 1/3 
-1 0 
1 4/27 5/54. —5/18 


X1 x2 x3 X4 XS5 X6 
1 
1 3/7 2/7 
1 17 -3/7 
1 1/14 -3/14 


1 


equaling to f—, and hence is optimal solution to the ILP problem, i.e., 
x* = (0,1, 2)!, Pmt. 


The corresponding enumeration tree is given in Fig. 11.5. 
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Fig. 11.5 Enumerate tree to (1) 
ILP problem (11.5.1) 


ty => 2 ri <1 
(3) (4) 
fy =2 T2=0 zi =0 wj=1 
3-1 f=15 e=2 ft=7 


11.5.2 Breadth-Oriented Strategy 


Each time when facing a pair of branches (LP subrelaxations), this strategy solves 
both of them and takes a branch as active that has the minimum objective value, 
say f, among the solved branches (excluding fathomed ones). It is obvious that the 
value 


f-="fl 


is an integer lower bound on the optimal value of the ILP problem. The strategy 
adds the objective cut 


cx> fo 
to the active branch and solves it. If the obtained optimal solution is integer, it is also 
optimal for the ILP problem. Otherwise, a valid pair of integer cuts are added to the 
active branch to yield a lower level pair of branches; and so on, until the optimal 
solution of a branch is integer as well as attains the current integer lower bound f. 
A branch is fathomed in one of the following cases: 


(i) Current branch is infeasible. 
(ii) The solution to the current branch is an integer. 
If it is a valid solution attaining f~, the solution is optimal to the ILP 
problem. 
Otherwise, if f is strictly less than the current upper bound on f*, update 
f+ = f as well as the corresponding objective cut. 
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(iii) The solution is noninteger and it holds that" fis greater than and equal to f v 
If the preceding condition is not fulfilled, update f~ = " f7, and continue. 


A tight integer lower bound on the optimal value of the ILP problem is certainly 
desirable. Since the optimal value of Example 11.5.1 is just equal to f~ =" f", it 
was solved in few iterations. Unfortunately, this is not the case in general. 

The following example would allow us to learn something about how the strategy 
performs with an ILP problem that has its optimal value far away from expected. 


Example 11.5.2 Solve the following ILP problem by the controlled-branch method 
with breadth-oriented strategy: 


min f = —7x; —5x24+ 9x3 
s.t. 3x, — x2 + 4x3 

xX) + 7x2 + 5x3 

integerx; > 0, j = 1, 2,3. 


a 


IA IA 
ee) 
n 


Answer Introduce slack variables x4, x5 to the inequalities. The initial tableau is 


xX] X2 x3 x4 \X5 | RHS 
3 -l 4 1 6 
1 7 5 1 35 
—7 -5 9 


The optimal simplex tableau is 


() 
xX] x2 X3 X4 X5 RHS 
1 3/2, 7/22, 1/22 | 7/2 
1 1/2 -1/22 3/22 | 9/2 
22 2 1 | 47 


(2) Since f = —47, we do not add f > f~ = f, but add x; < 3. 


X} x2 X3 X4 x5 x6 | RHS 

1 3/2 7/22 1/22 7/2 ; 
1 1/2 -1/22 3/22 9/2 

1 1 ) 


22 2 1 47 
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the optimal tableau to which is 


1 5/7 


1 1/2 


x4 x5 


1/7 
1 1/7 
5/7 


X4 
7/22 
=1/22 


x1 X23 X4 
1 3/2 7/22 
1 1/2 —1/22 
3/27/22 
22 2 
(4) Adding f > f~ = —43 to (2) gives 
x1 x2 X3 x4 x5 
1 
1 5/7 1/7 
33/7 1 1/7 
7 5 —9 
88/7 5/7 


X6 
1 
=17 
99/4 
44/7 


1/22 
3/22 


1/22 

3/22 

1/22 
1 


321 
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the optima simplex tableau of which is 


x] x2 X3 xX4 X5 X6 X7 RHS 
1 1 3 
1 9/88 —1/2 5/88 199/44 
1 -1/8 -11/2 3/8 5/4 
1 5/88 1/2 —7/88 3/44 
1 43 


x] x2 x3 X4 x5 X6 x7 RHS 
1 1 3 
1 9/88 —-1/2 5/88 199/44 
1 4 
DD sR" 21172 378 5/4 
1 5/88 1/2 —7/88 3/44 
1 43 
the optimal simplex tableau to which is 
Xt x2 x3 X4 X5 X6 x7 xg RHS 
1 1 3 
1 1 4 
5 1 —1 -—7 4 
1 3 1 1 
-9 —7 1 —5 2 
9 7 5 41 
(6) Adding x2 > 5 to the end tableau of (4) gives 
x] x2 x3 X4 x5 X6 Xx7 RHS 
1 1 3 
1 9/88 -1/2 5/88 199/44 
~1 =5 
{i 78. =11p- «378 5/4 
1 5/88 1/2, —7/88 3/44 


1 
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Fig. 11.6 Enumerate tree to (1) 
ILP problem (11.5.2) ET? e209 
3 = f=-47 
m<3 t>4 
(2) (3) 
Tj) =3 F = 32/7 Infeasible branch 
z3=0 f= -307/7 
f > -48 
(4) 
t1=3 Z2 = 199/44 
3 = 3/44 f =—43 
wm <4 rw >5 
(5) (6) 
zi=3 2=4 £,=0 gg=—5 


=O f*=—-41|/%3;=-0 f=—-25 


whose optimal simplex tableau is 


Xi x x3 X4 X5 x6 7 xg RHS 


1 1 7 0 
1 -1 5 

5 -1 1 27 3 

iit £ 3 80) “a 

—44 -7 1 44} 18 

44 Zi 44 | 25 


The corresponding enumeration tree is given in Fig. 11.6. 


11.6 ILP: with Reduced Simplex Framework” 


Firstly, each of the ILP methods presented previously solves the LP relaxation of 
the ILP problem by the simplex method. If the solution is noninteger, it solves a 
series of LP relaxations (branches) by the dual simplex method. Since each LP 


2 The reader is referred to Sect. 18.1, Part II. 
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relaxation yields from its predecessor by adding a cut, the number of constraints 
increases in the solution process, as is unfavorable for solving large-scale ILP 
problems. 

This defect can be eliminated if the branch-and-bound method is actualized with 
the generalized simplex framework, which adds an integer cut by simply changing 
the associated bound. But the latter is not very relevant to the controlled-branch 
method. In this aspect, the generalized reduced simplex framework turns out to be 
desirable (see Sect. 18.1). Along this line, we will actualize the controlled-branch 
method with the breadth-oriented strategy. 

Consider the variable-bounded ILP problem in the reduced form below: 


min f, 
s.t. A (*) =b, |< integerx <u. at 


Let x be the current optimal solution to the LP relaxation of the ILP problem, 
corresponding to the objective value f = f~, where f~ is an integer lower bound 
on the optimal value of the ILP problem. 

We put the problem into the following form: 


xpanlf [F=f (11.10) 
ION \an41 XB 
which is termed reduced simplex tableau. 
Assume that solution x is feasible. If the bound violations defined by 
Lj, — Xj» if xj, < Lj, 
Pi = \ Uj, —X;j,, thx; > uj,, i=1,....m+1, (11.11) 
0, if Li = X ji SUj;, 
are all zero, then x attains f = f~ for the current branch. Since a branch is 


generated from the solved predecessor by adding a cut, only a single bound is 
violated at first. 
To handle a branch, we put forward the following subalgorithm in tableau form. 


Subalgorithm 11.6.1 (LP Subalgorithm: Tableau Form) Initial: Integer lower 
bound f~ on the optimal value and f = f~. A reduced tableau in form (11.10), 


with basic solution (x, f). This procedure seeks a solution, attaining f~, to the 
branch. 


1. Select row index 
p € argmax{|p;| |i =1,...,m+ 1}, 


where ; is defined by (11.11). 
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2. Return if pp = 0. 
3. Go to step 8 if 


J={j €T | sign(p)ap ; < O} U{j € IL | sign(pp)ap ; > O} = G, 


where I and IT are defined by (18.9) 

4. Determine column index q € arg maxjey |@p ||. 

5. Convert a), to 1 and eliminate the other nonzeros in the column by elementary 
transformations (without touching the xg column except for its pth component 
replaced by the value of x,). 

6. Update xg = xp — PpAjys X jp = Xj, + Pp. 

7. Go to step 1. 

8. Compute a = —Pp/Gp,n41. 

9. Return if a < 0. 

10. Ifa = 0, set a = 1; otherwise, seta ="a’. 
11. Add —a times of f column to ¥g column, and set f = f +a. 
12. Return. 


The preceding Subalgorithm has the following three exits: 


Step 2: The end x is a valid solution. If it is an integer, the solution is optimal to the 
ILP problem. If it is noninteger, execute the Subalgorithm on a deeper branch, 
yielded from adding an integer cut. 

Step 9: The branch is infeasible, and hence fathomed. 

Step 12: There is no valid solution to the branch. The end x is optimal to the branch 
with objective value f > f-. 


Assume that an integer valid solution is not attained in the process eventually. 
If all the branches are infeasible, so is the original ILP problem. In the other 
case, determine the solution associated with the minimum objective value, say f, 
among end objective values of the branches, excluding fathomed ones. If it is an 
integer, the solution is optimal to the original ILP problem; otherwise, update the 
objective cut with f— = ' f” to the branch, and execute the Subalgorithm, and 
so on, until an optimal solution to the ILP problem is found, or infeasibility is 
detected. 

The following is an illustrative instance: 


Example 11.6.1 Solve the following ILP problem: 


min f = —5x; + 8x2 — x3, 
S.t. 4x; — 5x2 + x3 + x4 =5; 
11.12 
—3x, + 7x2 — 2x3 + x5 = 2, ( ) 


0 < integer xj <6, j=1l,...,5. 
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Answer 


(1) Initial tableau: 


™~ SI 


Iteration |: 

Convert the preceding to reduced simplex tableau by pivoting on the entry in 
row 3 and column 1, and call the generalized reduced simplex Algorithm 18.1.2 
(Part II), yielding an optimal reduced simplex tableau to the LP branch below: 


23/4 —25/4 
0 —oo 


™~ 


(2) Change the lower bound on the objective variable f to the suspected- 
optimal value —6 in the tableau. Increase the objective value to —6 by 
giving increment 1/4, and accordingly update the solution by xg = 
(023/45/4) — (1/4)(—4/53/51/5)? = (1/5, 28/5, 6/5)". 


™~ BI 


Call Subalgorithm 11.6.1. It returns from step 2 with a feasible tableau with the 
objective value f = —6. 
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(3) As 6/5c, > 1/5c4 = 28/5c5 = 0, select x; as the cutting variable (1 < x; = 
6/5 < 2). Add integer cut xj > 2 by changing the lower bound on x, to 2 
(while leaving the branch with x; < | pending). 

Call Subalgorithm 11.6.1. 
Iteration 2: 


l.p =3,93 =2-6/5=4/5. 

3. J = {2}. 

4.q =2. 

5. Multiply row 3 by —5/8, and add —7/5, —11/5 times of row 3 to rows | 
and 2, respectively (without touching xg columns, but its third component is 
replaced by value, 0, of x2). 

6. xp = (1/5, 28/5, 0)? — (4/5)(7/8, 11/8, —5/8)? = (—1/2, 9/2, 1/2)". 


x1 x2 x3 x4 x5 f f=-6 
7/8 3/8 1 —5/8 —1/2 
11/8 —6/8 1 7/8 9/2 
—5/8 1 —1/8 —1/8 1/2 
u 6 6 6 6 6 oo 
x 2 1/2 0 —1/2 6/2 —6 
1 [2] 0 0 0 0 [—6] 


Iteration 3: 1. p= 1, 1 = 1/2, jp = 4. 


3.5 =9. 
8.0 = —(1/2)/(—5/8) = 4/5. 
10.0 = 1. 


11. %g = (—1/2, 9/2, 1/2)? — (—5/8, 7/8, —1/8)? = (1/8, 29/8, 5/8)'; 
f=-6+1=-5. 
Returned from step 12. Leave the branch pending. 
Handle the pending ILP subprogram by adding cut x; < | to the end tableau of 
(2). Change the upper bound on x; to 1. 
Call Subalgorithm 11.6.1. 
Iteration 4: 


(4 


~m 
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5. Multiply row 3 by 5, and add —1/5, 7/5 times of row 3 to rows | and 2, 
respectively (without touching the xg columns but is the third component 
replaced by the value, 0, of x3). 

6. XB = (1/528/50)" + (1/5)(—1, 7, 5)? = (0,7, 1)T. 


= 


™~ SI 


Iteration 5: 


l. p =2,p2=6-7=-1, jy =5. 
35 = 


10.@ =1. 
11. xg = (0,7, I)? — (-1,2, D7? =(1,5,0)?, f=-6+1=-5. 
Returned from step 12. The ILP subprogram is stalled. 


~ as 


It is seen that (3) and (4) give the same lower objective bound —5. We finally grasp 
the end solution of (4) since it is a feasible integer, and hence optimal to the original 
ILP problem, i.e., 


x* = (1,0,0,1,5)', f* =—5. 
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It is imageable that more iterations would be required if one uses some “bad” 
integer lower bound on the optimal value. See that following instance with binary 
(0 or 1) variables. 


Example 11.6.2 Solve the following 0-1 ILP problem: 


min f = 4x; + 2x2 + 3x3 + 3x4 4+ 2X5, 
s.t. 3x1 — 4x2 — 7x3 — 3x4 < 
x1 + 2x2 — x3 —X4 — X5 < 
—3x2 + 1lx3 — 3x4 - 6x5 < 
xj €{0,1}, jgoHl,...,5. 


Answer Replacing x; € {0,1} by 0 < integer x; < 1, the 0-1 problem is 
converted to a normal ILP problem, to which the controlled-branch method applies. 


Introducing slack variables x;, j = 6,7, 8, solve its LP relaxation. 
Initial tableau: ¥ = (0, 1, 0,0, 1,2,0,8)?, %9 = 4 


aon]ss 


= 


Ser Sst 


Converting the preceding tableau to reduced one by multiplying row 4 by 1/2, 
and adding 4, —2, 3 times of row 4 to rows 1,2,3 gives the following tableau (without 
touching the last column—only its fourth component changed accordingly): 


rFPwoonl!ss> 


= 


SSeS 
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(1) Call generalized reduced simplex Algorithm 18.1.2. 
Iteration 1: 


l.a=1, p=1. 

2. xp = (2,0, 8, 1)7 +1 x (-2, 1, —3/2, —1/2) = (0, 1, 13/2, 1/2)", x =4— 
1=3. 

4.q=5 

5. Multiplying row 1 by 1/4, and adding 3, 3, —1 times of row 1 to rows 2,3,4, 
respectively 


(without touching the xg column (the first entry is replaced by the value 1 of x5)). 


Iteration 2: 


l.a=2, p=1. 

2. Xp = (1, 1, 13/2, 1/2)" + 2 * (—1/2, —1/2, —3, 0) = (0, 0, 1/2, 1/2)', x9 = 
g3—2= 1; 

4.q =3. 

5. Multiplying row 1 by —4, and adding 19/4, —59/4, —7/4 times of row | to rows 
2,3,4, respectively 


(without touching the xg column but its first component replaced by the same value, 
0, of x3). 
The optimal value of the LP relaxation is strictly greater than 0. Therefore, the 


current solution attains the integer lower bound, f = 1, on the optimal value of the 
ILP problem. So, add objective cut x9 > f~ = 1. 


(2) Add cut x2 > 1 and leave x2 < 0 pending. 
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Call Subalgorithm 11.6.1 (f~— = 1, x2 > 1). 
Iteration 3: 


1. p=4,04 =1-—1/2=1/2, jp =2. 
3. J=9. 
8. a = —(1/2)/(—7/2) = 1/7 > 0. 
10.a0="1/7'=1. 
11. xg = (0,0, 1/2, ij) — 1 x 2,9, —65/2, =T/25" = (—2, —9, 33,4)", % = 
14+1=2>f°-=1. 
This branch is pending. 


(3) Handle the pending branch, created by adding x2 < 0. 


Call Subalgorithm 11.6.1 (f~— = 1, x2 < 0). 
Iteration 4: 


.p=4,p4=0-1/2=—-1/2, jp = 2. 

. J = {1,4,5, 6}. 

. q = 1, max{|37/2], |6], |7|, |[3/2|} = 37/2. 

. Multiply row 4 by 2/37, and then add 11,47, —353/2 times of row 4 to rows 
1, 2, 3, respectively (without touching the xg column, but its fourth component 
is replaced by value, 0, of x1). 

6. xg = (0,0, 1/2, 0)'+(1/2)(22/37, 94/37, —353/37, 2/37)? = (11/37, 47/37, 

—158/37, 1/37)", x2 = 1/2+ (-1/2) =0. 


an B&B We 


22/37 21/37 6/37 —4/37 
94/37 28/37 —45/37 —7/37 
~353/37 —342/37* —288/37 44/37 


2/37 12/37 14/37 3/37 
[0] 1 1 +00 +00 LOO 
0 11/37 0 0 47/37 —158/37 
0 0 0 0 0 


= 


™ 8 
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Iteration 5: 


. p = 3, p3 = 0 — (—158/37) = 158/37, jp = 8. 

. J = {4,5}. 

. g =4, max{| — 342/37], |6|, | — 288/37|} = 342/37. 

. Multiply row 3 by —37/342, and then add —21/37, 28/37, —12/37 times of row 
3 to rows 1, 2,4, respectively (without touching the xg column but its second 
component is replaced by value, 0, of x4). 

6. gp = (11/37, 47/37, 0, 1/37)? — (158/37)(7/114, —14/171, —37/342, 2/ 

57)! = (2/57, 277/171, 79/171, —7/57)", xg = —158/37 + 158/37 = 0. 


nA BWR 


1/114 6/19  —2/57 7/114 
568/171 —11/19 —49/171 1 -14/171 
353/342 16/19 —22/171 


1 —16/57 2/19 7/57 
+OO 

0 2/57 79/171 0 0 277/171 

0 0 0 0 0 0 


Iteration 6: 


l. p=4, p4 = 0 — (—7/57) = 7/57, jp = 1. 

3. J=9%. 

8. a = —(7/57)/(—3/19) = 7/9 > 0. 

10.0 = 77/9? = 1. (11) xg = (2/57, 277/171, 79/171, —7/57)' — 1 x 
(—1/38, 2/57, —11/114, —3/19)' = (7/114, 271/171, 191/342, 2/57)', 
xXo=14+1=2> f-=1. 

The branch is pending. 


Comparing the end objective values reached in branches (2) and (3) gives a new 
integer lower bound, f~ = 2, on the optimal value of the ILP problem. 


(4) We continue to handle (2) by adding objective cut x9 > f~ = 2 first. 


= 


™ eI 
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Call Subalgorithm 11.6.1 (f~ = 2, x2 > 1). 
Iteration 7: 


.p=4, p=1-4=—3, jp =2. 

. J = {1,4,5, 6}. 

. gq = 1, max{|37/2], |6], |7|, |[3/2|} = 37/2. 

. Multiply row 4 by 2/37, and then add 11,47, —353/2 times of row 4 to rows 
1, 2, 3, respectively (without touching the xg column but its second component 
is replaced by value, 0, of x1). 

6. Xp = (—2,—9,33,0)' + 3(22/37, 94/37, —353/37, 2/37)' = (—8/37, 

—51/37, 162/37, 6/37)', x2 =4+(-3) =1. 


mn B&B We 


xo = 
22/37 21/37 6/37 —4/37 =8/37 
94/37 28/37 —45/37* —7/37 4/37 | 51/37 
—353/37 —342/37 —288/37 44/37 162/37 
12/37. 14/37. 3/37 6/37 


+OO +OO +OO 
0 —51/37 162/37 
0 0 


Iteration 8: 


. p= 2, max{|0 — (—8/37)|, |0 — (-51/37)|} = 51/37, p2 = 51/37, jp =7. 

. J = {4, 5, 6}. 

. g = 5, max{| — 28/37], | — 45/37], | — 7/37|} = 45/37. 

. Multiply row 2 by —37/45, and then add —6/37, 288/37, —14/37 times of row 
2 to rows 1,3, 4, respectively (without touching the xg column but its second 
component is replaced by value, 0, of x5). 

6. XB = (—8/37, 0, 162/37, 6/37)? — (51/37)(2/15, —37/45, —32/5, 14/45)? = 

(—2/5, 17/15, 66/5, —4/15)", x7 =0. 


np We 


—2/15* 2/15 
7/45 —37/45 
12/5 32/5 
1/45 14/45 


0 17/15 
0 
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Iteration 9: 

1. p=1, max{|O—(—2/5)|, |1—(17/15)|, |O—(—4/15)|} = 2/5, p1 = 2/5, jp = 
3. 

J = {6}. 

q=6. 


. Multiply row 1 by —15/2, and then add —7/45, —12/5, —1/45 times of row | to 
rows 2, 3, 4, respectively (without touching the xg column but its first component 
is replaced by value, 0, of f). 

6. xp = (0, 17/15, 66/5, —4/15)? — (2/5)(—15/2, 7/6, 18, 1/6)' = (3,2/3, 

6, —1/3)', x3 =0. 


1. p=4, pg =1/3, jp = 1. 

3. J=%. 

8. a = —(1/3)/(—1/6) = 2 > 0. 

10. a =2. 

11. xg = (3,2/3, 6, —1/3)? — 2(1/2, —1/6, —1, —1/6)' = (2,1,8,0)', x» = 


2+2=4> f~— =2. This branch is pending. 
(5) We continue to handle (3) by adding objective cut x9 > f~ = 2. 


1/114 =6/19 =2/57 7/114 
568/171 —11/19 —49/171 1 -14/171 


353/342 16/19 —22/171 

1 —16/57 2/19 7/57 
1 1 1 +OO +OO 
x|2/57 0 7/114 191/342 0 0 271/171 
0 0 (0) 0 0 0 


Call Subalgorithm 11.6.1 (f~ = 2, x2 < 0). 
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Iteration 11: 


1. pj = 0,7 = 1, 2, 3, 4. Returning from step (2) gives a valid solution: 
j = argmax{4(2/57), 3(7/114), 3(191/342)} = 3. Add cut x4 > 1 and let 
x4 < 0 pend. 


Call Subalgorithm 11.6.1 (f~ = 2, x2 < 0, x4 > 1). 
Iteration 12: 


. p=3, p3 = 151/342, jp = 4. 

. J = {6, 8}. 

. gq =6, max{| — 22/171], | — 37/342|} = 22/171. 

. Multiply row 3 by —171/22, and then add 2/57, 49/171, —7/57 times of row 
3 to rows 1, 2,4, respectively (without touching the xg column but its third 
component is replaced by value, 0, of f). 

6. Xp = (7/114,271/171, 0, 2/57)? — (151/342)(—3/11, —49/22, —171/22, 

21/22)" = (2/11, 113/44, 151/44, -17/44)", x4 = 1. 


On BW Re 


Xx] XQ 4X3 x4 x5 X6 x7 xg |Xo(f)| Xo =2 

ai... 37 a 1/1 Oo] 2/11 

45/44 —49/22 —27/11 1 7/44| 1/4) 113/44 

~353/44 M1722 it 1 37/44| 3/4| 151/44 

1 31/44 21/22 10/11 ~3/44* | 1/4] —17/44 

u 1 (o) t #1 1 +00 +00 +00/ +oof 
x|—17/44 0 2/11 1 0 151/44 113/44 o| 2 
1 0 0 0 [1] 0 0 0 0} [2] 


Iteration 13: 


1. p= 4, pP4 = 17/44, Jp =; 

3. J = {8}.(4)¢ =8. 

5. Multiply row 4 by —44/3, and then add —1/11, —7/44, —37/44 times of row 
4 to rows 1, 2,3, respectively (without touching the xg column but its fourth 
component is replaced by value, 0, of xg). 

6. Xp = (2/11, 113/44, 151/44, 0)? — (17/44)(4/3, 7/3, 37/3, —44/3)) = 
(—1/3, 5/3, —4/3, 17/3)", x; =0. 
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xy x2 X3 x4 x5 X6 X7 Xg | Xo(f) | Xo = 2 

4/3 2/3 n 1 2/3 1/73 | —1/3 

7/3 8/3 @ <1 1 178 5/3 

37/3 2/3 4. 14/3 1 =7/3:| =Aj3 

| —44/3 —31/3 -14 —40/3 1 11/3 17/3 
u 1 [0] 1 1 1 +OO +OO +OO +OO 
x 0 G =i 1 (48 58 175 2 
I 0 0 0 [q 0 0 0 0 [2] 


Iteration 14: 


l. p=3, p3 =4/3, jp =6. 

3. J =o. 

8. a = —(4/3)/(—7/3) = 4/7 > 0. 

10.@=1. 

ll. ¥g = (—1/3,5/3, —4/3, 17/3)? — 1 x (-1/3, -1/3, -7/3, 11/3)? = 


(0, 2,1,2)', % =2+1=3> f~ =2. 
This branch is pending. 


(6) The end objective value 3 reached by branch (5) is less than the value 4 by 
branch (4). So f~ = 3 is the new integer lower bound on the optimal value. 
Continue to handle (5) by adding objective cut f > f~ = 3. 


Call Subalgorithm 11.6.1. 


l. pj = 0, j = 1,2,3,4. Since it is integer, the valid solution is optimal to the 
original 0-1 problem. The optimal solution and optimal value are as follow: 


x* = (0,0, 0, 1,0), f* =3. 


The preceding instance is just for illustration of the controlled-branch method. 
In fact, it is simpler to solve the 0-1 problem otherwise by the following implicit 
enumeration approach: 

First, x = (0, 0, 0, 0, 0)" is ruled out to be optimal because it is infeasible. 

Then, compare objective values associated with solutions having only a single 
1-valued component, i.e., the coefficients, 4, 2, 3, 3, 2, of the objective function. 

Corresponding to value min{4, 2, 3, 3, 2} = 2, both solutions (0, 1, 0, 0, 0)? and 
(0, 0, 0, 0, 1)" are ignored since they are infeasible. 

Therefore, turn to (0, 0, 1, 0, 0)? and (0, 0, 0, 1, 0)", corresponding to the second 
minimum 3. It is then clear that the later is the only optimal solution to the 0-1 
problem. 
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A tight integer lower bound f~ on the optimal value of the ILP problem is 
conducive to the controlled-branch method; it is even desirable if the optimal value 
of the ILP problem is available. For some ILP problems, like the 0-1 problem, it 
would be possible to determine a tight integral lower bound on the optimal value 
beforehand. 
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Part II 
Advances 


Chapter 12 ®) 
Pivot Rule Cee 


Consider the following standard LP problem: 
min f =c!x (12.1) 
s.t. Ax =b, x >0, 
where A € R™*", cE R", bE R™, m <n, andrank (A) = m, unless indicated 
otherwise. 
The pivot rule! plays an important role in the simplex method. Starting from 
a vertex of the feasible region, the method moves from a vertex to vertex until 
reaching an optimal vertex. The path consists of a series of edges. In essence, it 
is the pivot rule that determines the number of edges, or the number of iterations 
required. Indeed, the pivot rule used embodies the soul of the simplex method. 
Having long been used in practice, Dantzig’s original rule selects a pivot column 
index g such that 


q € argminZ;. 
JEN 


In principle, however, any index associated with a negative reduced cost is eligible 
to be chosen. This provides room for improving the simplex method. 

As early as the beginning of LP, Dantzig noted that the original rule was far from 
ideal, and turned to a so-called most-improvement rule. According to (3.10) and 
(3.11), the objective decrement is 


f- f= —Zq(bp/apq)- 


' The pivot rule is referred to as the column pivot rule, as after a pivot column is determined, the 
choice of a row pivot is very limited. 
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The most-improvement rule selects q, p such that the preceding is the largest 
possible. Although this seems to be attractive, the associated computational effort 
is too high to realize. It turned out that this rule performed poorly (Kuhn and 
Quandt 1953; Jeroslow 1973) in practice. In fact, the objective decrement is not only 
dependent on Z, but also on the stepsize a. If a is too small, so is the decrement. In 
fact, the decrement would even vanish in the presence of degeneracy. 

It seems to be attractive to pursue an “ideal” pivot rule that determines the 
shortest path of all paths connecting a given initial vertex to an optimal one. 
Therefore, the pivot rule has received great attention in the past. Various pivot rules 
have been proposed from time to time. However, all efforts have failed, and it is not 
known whether there is a rule that turns the method to polynomial time. Practically, 
the method can stall at a vertex for too long a time before exiting it in the presence 
of degeneracy when the number of hyperplanes meeting at a vertex exceeds the 
dimension of the feasible region, let alone the cycling possibility. 

In fact, infinite rather than finite rules are widely applied in practice. It is 
essentially a practical issue to judge the merits of the rules: their vitality or value is 
determined by the performance only. 

In this chapter, therefore, we introduce some favorable pivot rules, ignoring their 
theoretical aspect. 

Assume that the current simplex tableau is 


I oN b (12.2) 


For simplicity, assume that the associated basis and nonbasis are as follows: 


B={l,....m}, N={m+1,...,n}. (12.3) 


12.1 Partial Pricing 


The conventional method to obtain the reduced costs used in pivoting is to solve the 
system 


B'y =cp, (12.4) 
for y first and then calculate the reduced costs (pricing), i.e., 


Zn =cn — N'5. (12.5) 
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Computing all components of zy by the preceding formula requires (n — m) x m 
multiplications, termed full pricing. Consequently, a large proportion of running 
time is spent on pricing by the simplex method, especially when n — m > m. 

Such a situation hastens some variants by a so-called partial pricing strategy, that 
is, only a portion of Zy is calculated in each iteration. 

Pricing options in MINOS (Murtagh and Saunders 1998) include full pricing, 
sectional pricing, and multiple pricing, the last two of which belong to the partial 
pricing category (see Benichou et al. 1977; Chvatal 1983; Maros 2003; Nocedal and 
Wright 1999). 


(A) Sectional pricing. 


As a typical partial pricing, it partitions all column indices into p (e.g., p = 20) 
sections of approximately equal number of indices. In order to treat each index 
equally, the m logical indices are almost equally distributed into the p sections. 

In each iteration, pricing starts with the section that follows the section from 
which the previous pivot column index is selected (excluding basic variables). If 
a reduced cost is found in the section that is lower than a dynamically changing 
negative threshold, the related index is selected to enter the basis; otherwise, pricing 
is continued similarly with the subsequent section, and so on. 

If all the p sections are priced and the minimum reduced cost is still greater than 
the threshold, optimality is achieved numerically if it is greater than the negative 
optimality tolerance (normally —10~°); otherwise, the index associated with the 
minimum reduced cost is selected to enter the basis, and the size of the minimum 
reduced cost is recorded. 

Subsequently, the threshold is decreased (e.g., to 0.1 times of itself) whenever 
the size is not sufficiently greater than (e.g., lower than 1.1 times of) the threshold. 
An initial value of the threshold is set to a very large value (e.g., 107°) by MINOS. 
So, full pricing is actually carried out in the first iteration, at least. 


(B) Multiple pricing. 


Firstly, it determines a small set, consisting of nonbasic indices, associated with 
negative reduced costs. Then a pivot index is determined by some rule, e.g., by 
choosing an index associated with the minimum cost among them. If the cost is 
lower than some dynamically changing negative threshold, enter the according index 
to the basis. In the next iteration, computed are only the reduced costs associated 
with the rest indices in the set, from which a pivot index is determined, and so on. 

When all reduced costs associated with the set are greater than the negative 
optimal tolerance, a new set of nonbasic indices is determined, and the preceding 
procedure is repeated. Optimality is achieved if no such a set is found. 

It is initially anticipated that partial pricing requires more iterations than full 
pricing. What is expected is only that the overall efficiency is improved due to 
the reduction of computational effort per iteration by partial pricing. Surprisingly 
enough, it turns out that the number of iterations as well as the overall running 
time required by partial pricing reduces in general (see, e.g., Harris 1973; Pan 1997, 
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2008). Extensive numerical experiments confirm the superiority of partial pricing to 
full pricing. This fact would suggest searching for better pivot rules. 


12.2 Steepest-Edge Rule 


In the second half of the eighties and the first half of nineties of the twentieth 
century, when interior-point methods climaxed, some scholars thought of such 
methods being superior to simplex methods for solving large-scale LP problems. 
During the same period, on the other hand, the simplex methods got active strength 
with computational results, reported on the steepest-edge pivot rule (Forrest and 
Goldfarb 1992). Consequently, the two types of methods inextricably involved in 
fierce competition (Nemhauser 1994; Bixby 2002). 

Recall the assumption that the current basis B consists of the first m columns of 
A. The n — m edge directions, emanating from the current vertex, are as follows: 


é —B-l,. 
= ( : 2), jEN, (12.6) 


ej-—m 


where é jm € R”~™ is the unit vector with its (j — m)th component equal to 1. For 
each j € N, the reduced cost 


Zj=cj- chB la; 
is equal to the inner product of the objective gradient c and edge direction d/, i-e., 
Zj=cid/, jeEN. 


Ifz; < 0, thend J is a descent edge direction. Thus, Dantzig’s original standard rule 
selects a column index 


q € argmin Z;, 
JEN - 


associated with a descent edge direction, along which the corresponding decrement 
of the objective value attains the maximum amount for a unit increment in the 
entering variable. 

The following pivot rule uses normalized reduced costs: 


Rule 12.2.1 (Steepest-Edge Column Rule) Select the column index 
q € argmin{zZ;/||d|| | j < N}. 


It is noted that the z ; /||d J\|, Vj € N, differs from the cosine of the angle between 
c and d/ by the same constant factor 1/||c||. Therefore, the preceding rule actually 
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selects a descent edge direction that forms the maximum angle with the objective 

gradient, which is why the rule is prefixed by the steepest-edge. While it seems to 

be attractive, computing the norms ||d/||, Vj € N, is too expensive to realize. 
Goldfarb and Reid (1977) offered a scheme, in which the ||d/||?, Vj € N, 

were calculated by recurrence formulas in each iteration. Further, they gave several 

variants of the rule and reported remarkable numerical results, obtained in extensive 

experiments. This work received wide attention from the academic community. 
The recurrence formulas can be derived based on the basic change, i.e., 


d? = —(1/o)d!, (12.7) 
di =di —(o;/oq)d’, jf EN, J #4, (12.8) 


where o; denotes entries in the pivot row, Le., 
oj =a;B ep, jEN, (12.9) 


which along with (12.7) and (12.8) gives the following formulas: 


la? \I? = A /og) lla", (12.10) 
ld |)? = IIa"? — 2@j/oq)ajv t+ (oj/oq)ld"\", FEN, J #4, 
(12.11) 
where 
Bag =ag, B'v=4y. (12.12) 


In fact, only the second system of the preceding is needed to solve because ag has 
already been calculated. Moreover, a, can also be employed to directly compute 


7? = 14 |lagll?. 


It is therefore seen that updating squares of norms of edge directions involves 3 
linear systems: besides the two typical systems, another system is 


Bh =e,. (12.13) 


Since the simplex algorithm also involves Bag = dg and (12.13), it is only needed 
to solve an additional system; that is, 


ieee 
Bov=4q. 
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Table 12.1 CPU time and ratio 


Dynamic Projective 
Pivot rule Dantzig Devex Steepest-edge steepest-edge steepest-edge 
Hours 110.19 7.76 4.90 4.08 3.89 
Time ratio | 28.33 1.99 1.26 1.05 1.00 


In (12.11), in addition, only those inner products ajv that correspond to nonzero 0; 
are needed to calculate. Fortunately, most of 0; often vanish in sparse computations. 

The steepest-edge rule requires more computational effort per iteration but fewer 
total iterations. Numerical results with the steepest rule are listed in Table 12.1 in 
the next section for a comparison with other rules. 

A shortcoming is that the rule has to compute initial squares of the norms 
of edge directions corresponding to all the nonbasic indices. According to the 
restarting strategy, this must be carried out periodically to avoid overaccumulation 
of the rounding error. Consequently, the rule involves a considerable amount of 
computational effort for large-scale problems, especially when it is not possible to 
take advantages of computer structure (like Forrest and Goldfarb 1992) to solve 
linear systems efficiently. Moreover, it is clear that the partial pricing strategy is not 
amenable in such a context. 


12.3 Approximate Steepest-Edge Rule 


In this section, the notation of the previous section will still be used to present three 
approximate variants of the steepest-edge rule. 


(A) Devex row rule. 


Enlightened on the graphic approach to solving LP problems of 2-dimension, 
Harris (1973) proposes an idea of searching along an approximate steepest-edge. 
The resulting rule, named Devex, is simple, involving the pivot row and column 
only, and performs well in computational experiments. 

Let a set of n — m column indices, termed “reference framework,” be given. 
Assume that the subvector d/ , J € N, consists of components, located on the 
reference framework; the weight t; is endowed to the reduced cost Z;, as an 


approximation of \Id j|. 


Rule 12.3.1 (Devex Row Rule) Select the pivot column index 
p € argmin{Z;/t; | j € N}, 


where the weights ¢; are determined as follows, so that the preceding rule may be 
regarded as selecting the steepest-edge under the reference framework. 
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Initially, the set of current nonbasic indices is taken as a reference framework. 
Set ¢; = 1 for all indices j in it. As a result, the Devex rule coincides with the 
Dantzig’s original rule in the first iteration. 

The weights ¢; are updated in subsequent iterations. It is clear that (12.7) and 
(12.8) are still valid if vectors involved in them are replaced by vectors, consisting of 
components located only on the reference framework. Thus, we have the following 
updating formulas: 


ip = max{1, ||d4\I/logl}. (12.14) 
7) = max{t;,|oj/og|lld7I}, JEN, i #4, (12.15) 


where the last formula comes from using the larger size of the vectors 
di and — (a; /oq)d4 
to replace the size of their sum 
di — (0; /oq)d!. 


Since dg is independent of the weight updating in each iteration, \|d4 || is easy to 
compute as d4 is available. When g does not belong to the reference framework, on 
the other hand, \\d4 || could become very small even though it holds that ||d7|| > 1. 
In this aspect, (12.14) ensures that all weights t; are no less than 1. It is therefore 
clear that their weights never decrease even if indices stay nonbasic for a long time. 

When the rounding error accumulates too much, it is necessary to determine 
a new reference framework and set all weights to 1 again. As the weight fj is 
calculated directly, it is convenient to monitor errors in the process, which should 
be restarted when the calculated value differs from the updated value by a relatively 
large margin, e.g., when the former exceeds a multiple of the latter.” 


(B) Projective steepest-edge rule. 


Another scheme is to directly use \Id J|| themselves rather than their approxima- 
tion t j. Since (12.7) and (12.8) are still valid if vectors d/ in them are replaced by 
subvectors d/, recurrence formulas for squares of d/’s norms can be derived from 
(12.10) and (12.11) by replacing d/ with di. Itis still possible to directly compute 


44 ||? = 6 + llag ll’, 


where 6 is equal to | or 0, depending on whether qg belongs to the reference 
framework or not. All a; involved in the formulas should be regarded as subvectors, 


2 Harris uses twice as much. 
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corresponding to the reference framework. Pivot rules using these formulas are 
termed projective steepest-edge rule. 


(C) Dynamic steepest-edge rule. 


A further variant of the projective steepest-edge rule is to expend the reference 
framework when resetting it, by adding the index of the current leaving variable 
to the reference framework if the index is not already in it. Therefore, only minor 
changes are needed in the recurrence formulas. The resulting pivot rule is named 
dynamic steepest-edge rule. 

We quote computational results from Forrest and Goldfarb (1992) as follows: 

There were 20 test problems involved in their tests, including all 14 Netlib 

standard test problems that involve more than 10,000 nonzeros and six larger and 
more difficult problems collected by the authors. CPU times required by five codes 
based on Dantzig, Devex, the steepest-edge, dynamic steepest-edge, and projective 
steepest-edge to solve all the 20 problems are listed in the first row of Table 12.1. 
Time ratios of the first four rules to the projective steepest-edge rule are listed in the 
bottom row. 
It is seen from the preceding that Dantzig’s conventional rule is the slowest while 
the projective steepest-edge rule is the fastest for solving these test problems: their 
total time ratio is as high as 28.33. However, the authors indicate that the tests are 
favorable to the last three rules because they take advantages of the structures of 
computer IBM RISC system/6000 in pricing and solving systems. These numerical 
results firmly establish big superiority of the steepest-edge rule and approximate 
steepest-edge rules, such as Devex, over the conventional rule. 


12.4 Largest-Distance Rule 


Like the steepest-edge rule or Devex rule, the so-called largest-distance rule is also 
based on normalized reduced costs although it is simpler and easier to realize. 
We explore the (primal) pivot rule from the dual side. Consider the dual problem 


max g = bly 
st. ATy <c 


to gain some insight. In fact, a negative reduced cost 
Zj =cj— a; B'cp 


implies that the current dual solution y = B 'cg violates the dual constraint 
aj y < c;. Entering the associated variable x; to the basis means forcing the violated 


dual inequality constraint to be satisfied as an equality (binding), i.e., ajy = Cj. 
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Therefore, it reasonable to enter the variable associated with the mostly violated 
dual constraint to enter the basis. 

It is not surprising that the conventional rule is unsatisfactory, as it takes the 
nonbasic variable associated with the most-negative reduced cost to enter the 
basis, but the variable does not necessarily correspond to the mostly violated dual 
constraint. In fact, the signed distance of y to the boundary aj y = c; is (Sect. 2.1 
of Part I) 


d =Z;/|lajll. 


Even if z; is the most negative, the absolute value of d may still be very small when 
||a; || is large, so that the current dual solution is not far from the boundary. 

The following rule selects the variable, associated with the mostly violated dual 
constraint, to enter the basis (Pan 2008a). 


Rule 12.4.1 (Largest-Distance Column Rule) Select the column index 
q € argmin{Z;/||a;|ll 7 ¢ N}. 


The preceding actually represents a class of pivot rules if ||a;|| is regarded as any 
norm of a; although the Euclidean norm seems to be preferable. 

We quote the associated computational results from Pan (2008a): 

The actualization of the largest-distance rule is easy. The norms |la;|l, j = 
1,...,n, of the columns of the coefficient matrix remain unchanged in solution 
process. If these norms are calculated at the beginning, the number of divisions 
required by the pivot rule will not exceed n — m in each iteration. It is even better to 
normalize the columns of A first, so that the largest-distance rule is just the same as 
Dantzig’s rule because any reduced cost Z; itself is equal to the signed distance of y 
to the boundary. 

Computational experiments were carried out on an IBM PC with Windows 
XP 2002 system, processor 1.86GHz, 1.00GB internal storage, about 16 digits 
precision, and visual Fortran 5.0 compiler. There were three codes involved: 


1. Devex: Devex rule 
2. SCL1: Euclidean norm largest-distance rule 
3. SCL2: oo-norm Largest-distance rule 


MINOS 5.51 was used as a platform for the preceding three codes: only the pivot 
rule is replace by the tested rules, respectively. Altogether, three sets of 80 large- 
scale sparse problems were tested (see Appendix C: Tables C.1, C.2, C.3): 


1. 47 (according to m + n) largest Netlib problems (SCRS8-STOCFOR3) 

2. All 16 Kennington problems (KENO7-KEN18) 

3. All 17 BPMPD problems larger than 500KB (in compressed format) (RAT7A- 
DBIC1) 


Table 12.2 gives iteration and CPU time ratios of these codes. 
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Table 12.2 Iteration and Devex/SCL2 | Devex/SCL1 | SCL2/SCL1 

time ratios : ; ; 
Problem Iters | Time |Iters | Time |Iters | Time 
Netlib(47) 0.26 | 0.33 | 0.29 | 0.34 1.09 | 1.03 


Kennington(16) | 1.75 |2.18 |4.95 |4.33 | 2.84 | 1.98 
BPMPD(17) 1.29 |2.33 | 1.88 |3.64 | 1.46 | 1.56 
Average(80) 1.00 |2.06 | 1.43 |3.24 | 1.43 | 1.58 


It is seen from the bottom row that the total iteration and CPU time ratios of 
Devex to SCLI are 1.43 and 3.24, respectively. Therefore, SCL1 outperformed 
Devex significantly. It is more important that the gap between their CPU times 
required are larger. SCL2 also defeated Devex with CPU time ration 2.06. Moreover, 
it can be expected that the largest-distance rule using partial pricing will perform 
even better. 

As for the two largest-distance rules, SCL1 outperformed SCL2 with iteration 
ratio 1.43 and time ratio 1.58. Therefore, the rule using Euclidean norm seems to be 
preferable. 

Besides the remarkable performance of the largest-distance rule, the author 
himself was surprised that it is relatively easy to realize: SCL1 and SCL2 were 
yielded from MINOS 5.51 by only inserting a single sentence for normalizing 
columns of the coefficient matrix in its scaling subroutine. 


12.5 Nested Rule 


The nested pivot rule may be classified into the partial pricing category. In particular, 
it would be somehow close to multiple pricing. The key difference from the latter 
is that the former focuses on, and eliminate the most stubborn nonbasic indices that 
do not fulfil the the optimality condition (within the dual tolerance). 

Let € > 0 be the dual feasibility tolerance. At the beginning of each iteration, 
a set Nj of nonbasic indices is given. Pricing is carried out on Nj to determine 
a column index, associated with a negative reduced cost by, e.g., the conventional 
column rule. If the associated reduced cost is less than —e, then the index is selected 
to enter the basis, and N, for the next iteration is generated from the current N; by 
including all indices, associated with reduced costs less than —e. In the other case, 
do the same thing with Nz = N\N}j; if there is still no reduced cost less than —e, 
then optimality is achieved. 

As an instance, the nested counterpart of the classical rule is described below 
(Pan 2008b). 


Rule 12.5.1 (Nested-Dantzig Rule) Given a dual tolerance «. Set N) = N and 
No = G. 


1. Goto step 4ifNi ={j © Ni |Z; < —e} F@. 
2. Go to step 4if N) ={j € N2|Zj < -E} FV. 
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3. Stop (optimality achieved). 
4, Select a pivot column index q € argmin{z; | j € Ni}. 
5. Update: Ni = Ni\q, No = N\N\. 


Such a rule is prefixed by nested since the current set Nj, which undergoes 
pricing with priority, is a proper subset of its predecessor N,. The idea behind it 
can be explained further as follows. 

In the initial iteration or iterations in which N2 is touched, a full pricing is carried 
out. After that, a series of nested pricings follow, which might be called a “circle.” 
As each Nj is a proper subset of its predecessor in a circle, the computational effort 
for pricing decreases monotonically iteration by iteration. In the kth iteration of a 
circle, moreover, the reduced costs associated with Nj are less than —e all the time. 
Therefore, it is reasonable to enter such a “stubborn” nonbasic index to the basis. 

In fact, the index set N; generated from the first iteration corresponds to 
nonnegative constraints, gradients of which form acute angles with the negative 
reduced gradient. Based on the most-obtuse-angle heuristic, one should try to make 
these constraints inactive; in other words, one should give indices associated with 
these constraints priority to enter basis. 

Any conventional pivot rule may be modified to a nested one. The nested 
steepest-edge rule can be obtained from Rule 12.5.1 by using 


q € arg min {Z;/||d/|| | j € Ni} 
in step 4 instead, and the nested Devex rule obtained by using 
q =argmin {Z;/t; | j € Ni}. 


We quote the computational results from Pan (2008c) below: 
The nested pivot rules are easy for actualization. Associated computational 
experiments involve the following five codes: 


. Dantzig: MINOS 5.51 (full pricing option) 

. Devex: Devex rule 

. P-Dantzig: MINOS 5.51 (default sectional pricing option) 
. N-Devex: Nested Devex rule 

. N-Dantzig: Rule 12.5.1 


nABWN 


These codes are all based on MINOS 5.51 as a platform, differing from the 
pivot rule used only. The software and hardware environments are the same as those 
described in the previous section, as well as the same 80 test problems (Appendix C: 
Tables C.1 (AFIRO-STANDMPS), C.2, C.3). Table 12.3 lists total iteration and CPU 
time ratios of the first four codes to the fifth (N-Dantzig). 

It is seen from the preceding table that the N-Dantzig and N-Devex, based on the 
nested pivot rules, are significantly superior to the classical ones, and N-Dantzig is 
the best. In fact, the N-Dantzig rule outperformed Devex rule by total iteration ratio 
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Table 12.3 Iteration and time ratios to N-Dantzig 


Dantzig Devex P-Dantzig N-Devex 
Problem Iters | Time Iters Time Iters | Time Iters Time 
Netlib(47) 5.16 | 5.95 1.20 1.21 4.65 4.00 1.04 0.95 
Kennington(16) 5.63 5.65 5.56 5.55 3.51 | 2.64 1.00 {0.91 
BPMPD(17) 8.29 12.86 3.83 | 6.54 5.04 5.20 1.18 1.22 
Average(80) 6.78 9.75 3.48 5.73 4.57 4.20 1.10 1.09 


3.48 and time ratio 5.73 and outperformed Dantzig’s rule by total iteration ratio 6.78 
and time ratio 9.75! 

Further computational experiments were conducted with nested pricing exten- 
sively. Contained in Appendix D, associated numerical results can be outlined below 
(see Table D.16): 

For 77 test problems, the nested-Dantzig rule defeated the steepest-edge rule by 
total time ratio as high as 25.22, even though the former required more iterations 
(with total iteration ratio 0.34). Even the largest-distance rule was not comparable 
with the nested-Dantzig rule. However, the nested steepest-edge rule was inferior 
to the steepest-edge rule, since the former restarted from scratch too many times. It 
should be pointed out that these tests were somehow unfair to steepest-edge rules, as 
they neither use recurrence formulas (thus the additional two systems were solved 
in each iteration) nor take the advantage of the computer’s structure to solve systems 
more efficiently. 

In summary, nested pivot rules are superior to their conventional counterparts by 
a large margin (except for the nested steepest-edge rule), and the nested-Dantzig 
rule is superior to the nested Devex rule. 


12.6 Nested Largest-Distance Rule 


Largest-distance rules can be improved further by combining the nesting strategy 
(Pan 2008c). The resulting rule is as follows. 


Rule 12.6.1 (Nested Largest-Distance Column Rule) The same as Rule 12.5.1, 
except for its step 4 using the following instead: 
q € argmin{Z;/||a;|ll 7 € N}. 
Computational results (Pan 2010): 
The following three codes were involved: 


1. Devex: Devex rule 
2. LDN1: Nested largest-distance rule with Euclidean norm 
3. LDN2: Nested largest-distance rule with oo norm 
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Table 12.4 Iteration ratio and time ratio 


| Devex/LDN2 | Devex/LDN1 | LDN2/LDN1 
Problem ‘Iters  |Time —|iters~—s| Time “liters —«| Time 
Netlib(47) | 1.20 1.17 | 1.15 1.16 | 0.96 0.99 
Kennington(16) ‘| 5.63 [5.42 [7.83 5.77 =|1.39 | 1.06 ~ 
BPMPD(17) | 4.32 | 7.43 5.43 10.17 | 1.26 1.37 
Average(80) | 3.69 | 6.08 4.34 7.27 | 1.18 1.20 


The software and hardware environments are the same as those given in the 
previous two sections, with the same 80 test problems (Appendix C: Tables C.1, 
C.2, C.3). Table 12.4 lists total iteration and CPU time ratios. 

It is seen that the nested largest-distance rules outperformed Devex rules 
unambiguously in terms of either iterations or CPU time. As for the two nested 
ones, the nested largest-distance rules with Euclidean norm is better, as defeated 
Devex rule by iteration ratio 4.34 and time ratio 7.27. The margins are even larger 
than that between the nested-Dantzig rule and Devex rule. 

In view of the numerical results reported in Sects. 12.4—12.6, we conclude that 
the nested largest-distance rules with Euclidean norm is the best among existing 
pivot rules, at least in terms of the 80 tested problems. 
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Chapter 13 ®) 
Dual Pivot Rule hac 


A pivot rule used in the dual simplex method is termed dual pivot rule.'! Like in 
the primal simplex context, a dual pivot rule plays an important role in the dual 
algorithm. This chapter will present very promising dual pivot rules. 

Assume that x = B~'b < 0. The dual Dantzig’s rule selects a row index 
p € B such that x, is the minimum among components of the basic solution. Thus, 
the dual objective value will increase the most possible for a unit stepsize. This 
rule is far from ideal, like its primal counterpart. The so-called most-improvement 
rule determines the pivot such that the corresponding dual objective increment 
attains the maximum possible in the current iteration. Unfortunately, this rule is 
still unsatisfactory, just as its primal counterpart. 

It might be well to assume that the current basis matrix consists of the first m 
columns of A, thus 


B=({l,...,m}, N={m+1,...,n}. 


13.1 Dual Steepest-Edge Rule 


Consider the dual problem 


max g=b'y, 
s.t. Aly <c. 


' The rule is termed “row” pivot rule since after a pivot row is determined, there is a very limited 
choice of a pivot column. 
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Let y be the current dual basic feasible solution, satisfying 


B'S =cg, (13.1) 
N'Y < ey. (13.2) 
Define 
y(B) = ¥— Bh, 
hi= B'e;, i=1,...,m. 
From the preceding two expressions and (13.1), for any i € {1,...,m} and B > 0 
it holds that 
a; y(B) =a) ¥ - B= -B Sci, 
ajy(B) =azy=cy, k=1,....m, k¥i. 
It is known that —A’, i = 1,...,m, isan edge direction, emanating from the vertex 


y of the feasible region {y|ATy < c}. The determination of row index i implies 
that the basic variable x; leaves the basis, and hence the constraint a} y < cj can be 
satisfied as a strict inequality. Since 


—b'h' = —e/ B~'b = —5j, 


when x; < 0, the edge direction —h' forms an acute angle with the dual objective 
gradient b, as is an uphill direction. Therefore, the objective value will never 
decrease if a row index p ¢€ {1,...,m} is selected such that x, < 0; it strictly 
increases if dual nondegeneracy is assumed. In particular, the following rule selects 
the edge that forms the smallest angle with b (Forrest and Goldfarb 1992). 


Rule 13.1.1 (Dual Steepest-Edge Row Rule) Select the pivot row index 
p € arg min{x;/||h'|| |i =1,..., m}. 
Like in the primal context, practicability of the preceding rule lies in computing 
|ni?, i=1,...,m, 
in a recurrent manner. 
Assume that xp is selected to leave and x, to enter the basis. Then the inverse of 


the basis matrix is updated by (Sect. 3.3 of Part D: 


= TBO 
_; (q-epley 
=B  ——SS 3 
4 pg 
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where dg = B "dy; Premultiplying the preceding expression by e} and transposing 
it lead to the recurrence for edge directions, i.e., 


A? — (1/dpq)h?, (13.3) 
Ai = h' —Gig/G@pq)h?, i=1,...,.m; is p, (13.4) 


from which the recurrence formulas for squares of norms of edge directions follow: 


AP ||? = (1/Gpq)" Wh? I’, (13.5) 
ai? = Whi? — 2Gig/Apq)ui + Gig/Gpq) WPI, 
i=1,....m,iF p, (13.6) 
where 
BThP=ep, Bu=h?. (13.7) 


Note that ||/?||?7_ = (h?)'h? can be directly calculated, because h? and Ag are 
obtained otherwise independently. 

Computations, involved in Rule 13.1.1, are usually cheaper than that in the 
(primal) steepest-edge rule. They both solve an additional system (13.6), hence 
Bu = h?. It is noted that formula (12.11) needs to solve B'v = dq. In contrast 
to (12.11), which is expensive when the pivot row e! B~!N is dense, there is no 
any inner product involved in the middle term of (13.6). In practice, it is often the 
case when n — m > m, in which initially computing squares of norms of edge 
directions in the primal rule is cumbersome, compared with those in the dual rule. 
As for taking advantage of sparsity, handling B~!a j is also inferior to B Te: 

Now let us turn to the dual problem of form 


max g=b'y, 


13. 
s.t. Aly+z=c, z>0. oe 


A new pivot rule can be derived if the steepest-edge direction is considered in (y, z)- 
space. 
It is clear that the dual basic feasible solution 
(3, Zn, ZB) = (B-"cp, cy — N'B™ "cp, 0) 


is actually the unique solution to the (m +n) x (m+n) system 


Br 


OT y CB 
N' I 0 Zn J = {en 1], 
0 OT ZB 0 
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which is, geometrically, a vertex of the polyhedron 
{(y,z) ER" x R"| Aly +z2=c,z > O}. 
The inverse of the coefficient matrix of the system is 
BY 0 -—BTtT 


—N™B-T 7 N™BT 
0 0 I 


It is easy to verify that the last m columns of the preceding are just edge directions, 
emanating from the vertex, i.e., 


—B-T 
hi'=| NBT )e, i=1,...,m. (13.9) 
‘4 


Recurrence formulas of these edge directions are of the form (13.3) and (13.4), and 
squares of norms of them are of form (13.5) and (13.6), though (13.7) should be 
replaced by: 


n 
Bth=ep, Bu=h+ )° ojaj, (13.10) 
j=m+1 


where a; is the jth component of the pivot row vector, i.e., 
o = Alh. (13.11) 
As o is computed while pricing, the following quantity: 
|n? |? =h'h+olo (13.12) 


can be directly computed. 

In this book, the pivot rule, based on (13.5)—(13.7), is referred to as dual steepest- 
edge pivot rule I, and that based on (13.5), (13.6), and (13.10)-(13.12) referred to 
as dual steepest-edge pivot rule I. Associated numerical results will be given in 
Table 13.1 in the next section, in a comparison with according approximate rules 
there. 


Table 13.1 Total time and ratio 


| Steepest | Dynamic | Projective Steepest 
Dual pivot rule | Dantzig | Devex | -edge II | steepest-edge | steepest-edge | -edge I 
Hours 177.78 |67.43 |12.72 | 10.41 | 7.36 6.36 


Time ratio 27.95 10.60 | 2.00 | 1.64 | 1.16 1.00 


13.2 Approximate Dual Steepest-Edge Rule 359 
13.2 Approximate Dual Steepest-Edge Rule 


Harris (1973) derived a dual variant of the Devex rule, as may be regarded as an 
approximation of the dual steepest-edge rule II. 

She constructed a “reference framework,” a set of m indices of components 
of z. Assume that subvector hi consists of components, located on the reference 
framework, of edge direction hi, Weights s;,i = 1,...,m, are endowed to x; to 


approximate || hi ||. 


Rule 13.2.1 (Dual Devex Row Rule) Select the pivot row index 
p € argmin{x;/s;|i = 1,...,m}, 


where the weights s;, i = 1,...,m, are determined as follows, so that the rule 
can be regarded as an approximation of the steepest-edge rule under the reference 
framework. 

At the beginning, index set B is taken as the reference framework, and s; are 
set to 1 for alli = 1,...,m (see (13.9)). Hence, the dual Devex rule is just the 
same as the dual Dantzig rule in this case. Subsequently, s; are updated iteration by 
iteration. Assume that o is defined by (13.11) and G is the subvector consisting of 
the components located on the reference framework. As (13.3) and (13.4) are still 
valid when vectors in them are replaced by subvectors, consisting of components 
located on the reference framework, we have the following updates: 


Sp = max{1, ||6||/|apq|}. 


Si = max{s;, Gig /Apq\|\|El|}, i=1,...,.m,iF p, 


the last of which comes from using the larger norm of vectors hi and — (Gig /Apq)h? 
instead of the norm of their sum (see (13.9)). In each iteration, ag = B dg is 
computed separately. As a is available, ||¢ || can be computed directly. 

Similar to the (primal) Devex rule, when errors caused by repeatedly using the 
updating formulas accumulate too high, it is necessary to determine a new reference 
framework and set all weights to 1 again. As weight f, is calculated directly, it is 
convenient to monitor errors. It should be restarted when the calculated value differs 
from the updated value by a relatively large gap, e.g., when the former exceeds a 
multiple of the latter (Harris used double). 

When the error, caused by repeatedly using of the updating formula, accumulates 
too much, it is necessary to determine a new reference framework and set all weights 
to 1 again, just like the approximate steepest-edge rules. It is also possible to monitor 
errors: the process should be restarted when the calculated value of ||¢ || differs from 
the updated value sp by a relatively large margin. 

The so-called dual projective steepest-edge rule results from using \|hi || rather 
than s5;. This rule can also be obtained by modifying the dual steepest-edge rule 
under the reference framework. In fact, replacing hi in (13.5) and (13.6) by i leads 
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to recurrence formulas of IIA \| 


system 


, whereas u in (13.6) can be obtained by solving the 


n 
Bu= ) Ojaj;. 


j=m+1 


A further variant of the dual steepest-edge rule results from expending the 
reference framework whenever it is reset, that is, by adding the pivot column 
index to the current reference framework if it is not in it already. Accordingly, the 
recurrence formulas are modified slightly. Such a variant is called dual dynamic 
steepest-edge rule. 

We cite numerical results reported by Forrest and Goldfarb (1992) below. The 
hardware and software environments as well as 20 test problems were the same as 
those described for the primal case, presented in Sect. 12.3. Codes based on six dual 
rules: Dantzig, Devex, steepest-edge II, dynamic steepest-edge, project steepest- 
edge, and steepest-edge I were tested. Table 13.1 gives the total CPU times, required 
for solving all the problems, in the first row, and time ratios of the first five rules to 
the dual projective steepest-edge rule in the second row. 

It is seen from the preceding table that the dual Dantzig rule is the slowest, 
whereas the dual steepest-edge rule I is the fastest, with time ratio as high as 27.95, 
though the tests are favorable to the last four rules because they take advantages 
of the structures of computer IBM RISC system/6000 in pricing and solving 
systems. 

In terms of reported numerical results, the dual rules appear to be inferior to 
their primal counterparts slightly. However, Forrest and Goldfarb indicate that these 
data are unfair to the dual rules, because their primal codes were optimized, while 
the dual ones were not; the numbers of hours required by the last four codes in 
the preceding table should be reduced by 10% at least, though decrements of the 
dual Dantzig and Devex are not as so much. Therefore, the dual rules are actually 
efficient, compared to their primal counterparts. 

The steepest-edge rule and Devex approximates as well as their dual variants are 
superior over the standard rules with large margins and are widely used in many 
commercial codes, such as CPLEX. 


13.3 Dual Largest-Distance Rule 


This section derive a dual variant of the largest-distance rule (Sect. 12.4). The basic 
idea is to determine a leaving variable by finding an inequality constraint, mostly 
violated by the current solution in the reduced space. 
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Let B = {1,...,m}, N = A\B be the current basis and nonbasis of the standard 
problem (12.1). Assume that the associated simplex tableau is dual but not primal 
feasible, i.e.,b = xp 0. 

For some i € B, the inequality x; < 0 implies that the solution, xy = 0, in the 
reduced space violates the constraint 


b; — (w')"xy = 0, 
where b = B~'b and 
wi = N'BTe;. 


The signed distance from the point xy = 0 to the boundary b; — (w')'xy = 0 
is X;/||w'||, where any norm is allowed in principle although the Euclidean norm 
should be preferable. 

The dual Dantzig conventional rule is 


p € argmin{x;|i = 1,..., mb}, 


which does not necessarily correspond to the mostly violated inequality constraint, 
since the according distance would be very small when ||w? || is large, as the point 
Xy = O may not be far away from the boundary actually. Therefore, it should not 
be expected that this rule performs satisfactorily. 

If one determines a leaving variable that corresponds to the mostly violated 
inequality constraint, the following rule follows. 


Rule 13.3.1 (Dual Largest-Distance Row Rule) Select the pivot row index 
p € argmin{</|Jw' || [i= 1,...,m). 


Involving ||w’ ||, the preceding rule is cumbersome, compared with the (primal) 
largest-distance rule. Like in the primal context, however, || w! \|7, b= Wake, 
can be computed recursively, so that the rule would be still practicable. 

Consider the set of n-dimensional vectors 


ot = A'BTe,, i=l,...,m. 
Note that w! is an (n — m)-subvector of a! for any i = l,...,m. Assume that 


Xp leaves and xg enters the basis, then it is not difficult to derive the following 
recurrence relation: 


oF = (1/apq)o”, 


o' =o' — (Gig/Gpqg)o”, i=1,...,m; iF p, 
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where the pivot column ag = Ba, is available. Further, recurrence formulas of 
squares of their norms can be obtained, i.e., 


~ py2 = \2 2 
lo?" = /apq) Ilo? I’, 

~i 2 i 2 = fo iT 
Ila" I" = lo" |" — 2@ig/apq)o' o? 


+(Gig/@pq) lo? (?, i=1,...,m, ix p. 


Combining the preceding, ooh = e} B-!NoP, and ||o'||? = ||w’ ||? + 1 leads to 
the following required formulas: 


|? ||? = (1/apg)? I (w? ||? + 1) - 1, (13.13) 
[| w! ||? = |w! I? — 2Gig /Apg ui (13.14) 


+(Gig/G@pq) (lw? |F +), i=1,...,m,i¢ p, 


where 


n 
Bu= y Ojaj, 


j=m+1 


and o; is the jth component of a? (i.e., the (7 — m)th component of w?). Note 
that components of w? are available entries of the pivot row, and hence ||w? ||? = 
(w?)' w? can be calculated directly. 


13.4 Dual Nested Rule 


This section derives a dual variant of the nested rule (Sect. 12.5). Regarding the 
negative components of the primal basic solution as elimination target, it focuses on 
the most stubborn ones among them. 

Let € > O be the primal feasibility tolerance. At the beginning of an iteration, 
a set 1; of row indices is given the priority for associated basic indices to leave 
the basis. The row index in J, corresponding to the smallest component of xg is 
determined. If the component is less than —e, then the corresponding basic index is 
selected to leave the basis, and the J; for the next iteration is born from the current 
T, by including all its row indices, associated with xg’s components less than —e. 
In the other case, do the same as before with 7 = B\J,. If no basic component is 
less than —e, then optimality is attained. 

The following is a nested variant of Dantzig conventional dual rule. 
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Rule 13.4.1 (Dual Nested Row Rule: Dantzig) Given primal tolerance € > 0, set 
I, = Band l, = 9%. 


Go to step 4ifhh ={ieh | x1 <-e} #9. 

Go to step 4if } = {i € In | Xi < -E} FG. 
Stop (optimality achieved). 

Select a pivot row index p € arg min {x; |i € I}. 
Update: I} = I\\p, In = B\Nh. 


MAWNS 


In the first iteration or iterations in which J is touched, the preceding rule 
actually proceeds the same as the standard dual row rule. After such an iteration, 
a series of iterations with nested pivoting follow, as might be called a “circle,” 
where each J; is a proper subset of its predecessor. In the kth iteration of a circle, 
basic components associated with J; are less than —e all the time. It is therefore 
reasonable to select such a “stubborn” index leaving the basis. 

It is possible to turn any full dual rule to a nested version. The dual nested 
steepest-edge rule can be obtained from Rule 13.4.1 by using 


p € argmin {%;/|h'|| |i € 1} 
in step 4 instead, the dual nested Devex rule obtained by using 
p € argmin{x;/s; |i € 1y}, 
and the dual nested largest-distance rule by using 
p € argmin{x;/||w' || |i < 1}. 
In view of the performance of their primal counterparts, it might be expected that 
the dual nested rules and the dual nested largest-distance rules perform satisfactorily. 


The dual nested rules are easy to implement, fortunately. However, there is no 
computational experience at present. 
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Chapter 14 ® 
Simplex Phase-I Method crests 


The simplex method requires a feasible basis to get itself started. The so-called 
Phase-I procedure is for this purpose. 

The Phase-I method in Chap. 3 of Part I is only for textbooks and has not been 
used in practice at all since it rigidly determines the initial basis and expands the 
scale of the problem. This chapter will present selected practicable Phase-I methods, 
which are artificial variable free or involve a single variable only. 


14.1 Infeasibility-Sum Algorithm 


The auxiliary objective function of this method only involves variables, associated 
with negative components of the current basic solution. The method is widely used 
in practice since it is artificial variable free and is workable. 

Assume that an initial simplex tableau of form (3.6) is available, associated with 
basic solution 


Xp=b, xXy =0. (14.1) 
Introduce 
T={i=1,...,m|b; <0}, T={l,...,m}\. (14.2) 
Assume that J 4 J. Construct the following auxiliary problem: 


min — dar Xji> 
s.t. xp =D—Nxn, (14.3) 


xj;,20, ier; xnv>=O0, 


Jt 
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where the objective function is the negative sum of variables associated with 
negative components of the basic solution. Note that the constraint system of 
the auxiliary problem is the same as that of the original problem, except for the 
nonnegativity restriction only imposed on variables, associated with nonnegative 
components of the basic solution. 

It is clear that the solution (14.1) is feasible for the auxiliary problem. The 
associated feasible simplex tableau can be obtained by eliminating nonzero entries 
in the objective row by relevant elementary transformations. Consequently, the 
reduced objective function is 


— ) 0 xj, = wo + Zyxn, (14.4) 
iel 
where 
w=->d bb >0, Z=>) aj, FEN, 
iel iel 
where the “>” inequality is from (14.2). 


Theorem 14.1.1 (Infeasibility Test) [f the reduced costs of an auxiliary simplex 
tableau are all nonnegative, the original problem is infeasible. 


Proof Assume that zy > 0, and x > 0 isa feasible solution to the original problem. 
Substituting + into (14.4) gives the system, whose left-hand side is clearly less than 
or equal to zero, and, by wo > 0 and Zy > 0, the right-hand side is strictly greater 
than zero, leading to a contradiction. Therefore, the statement is valid. oO 


If there is a negative reduced cost, then any existing column rule applies. Assume 
that a column index g was determined such that 


a= > oy a (14.5) 
ie] 


Since there is no nonnegativity restriction on the current infeasible basic 
variables, the auxiliary problem could be unbounded even if the original problem 
itself bounded. In fact, this happens when J = @ or 


{i € Taig > 0} = 9. (14.6) 
Thus, the conventional row rule is not applicable. As a remedy, the rule is modified 
as follows. 
Rule 14.1.1 (Auxiliary Row Rule) Determine the row index p such that 


min{b;/dig\4ig > 0, i € 1}, if {i € Taig > 0} 4G, 


sie 2a . : (14.7) 
max{b;/dig|diq < 0,i € I}, in the other case. 


a = bp/apq = 
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Lemma 14.1.2 Assume that the selected column index q satisfies (14.5). Then 
rule 14.1.1 determines a row index p such that the simplex tableau, resulting from 
the according basis change, is feasible to the auxiliary problem. 


Proof First, it can be asserted that there is a row index ¢ € J such that a;, < 0 
since, otherwise, from 


dig>0, Viel, 


it follows that 

Y > dig = 0, 

iel 
which contradicts the assumption (14.5). Therefore, the second expression of (14.7) 
is always well-defined, hence so is the row rule. 

Take Gp, as the pivot. Executing according elementary transformations leads 
to a new simplex tableau, whose right-hand side can be expressed in terms of its 
predecessor’s entries (Part I), i.e., 

bi = bi — aig, ieliF¢p, 
er 
where a is determined by (14.7). It is easy to verify that these components are all 


nonnegative, hence the resulting simplex tableau is again feasible to the auxiliary 
problem. Oo 


Theorem 14.1.3 Assume the feasibility of the original problem and nondegeneracy 
throughout solution process. The number of basic infeasible variables strictly 
decreases in finitely many iterations. 


Proof It is seen from the proof of Lemma 14.1.2 that the second expression of 
(14.7) is still well-defined if “max” there is replaced by “min,” i.e., 


a} = bs/dsq = min{b; /djq\dig < 0,, € I} > 0. (14.8) 


If, in some iteration, the stepsize w is determined by the first expression of (14.7), 
and it holds that 


a> ay, (14.9) 
then the s row-indexed component of the new solution satisfies 
&j, = by — AAsq = 0, 


which implies that a basic infeasible variable becomes feasible, at least. 


368 14 Simplex Phase-I Method 


If, in some iteration, the second expression of (14.7) is used, then, for the new 
solution, not only the basic components, associated with all row indices i € J, are 
feasible, but so are basic components, associated with row indices 


se{iel |dig <4. 


This is of course favorable. 

Assume that the number of basic infeasible variables does not strictly decrease 
forever. In each iteration, there is always a negative cost since the original problem is 
feasible (Theorem 14.1.1), and the stepsize a is determined by the first expression 
of (14.7), which does not satisfy (14.9). But the number of bases is finite, hence 
there are some bases that appear infinitely, as cycling occurs. This is impossible 
because the objective value of the auxiliary problem strictly decreases under the 
nondegeneracy assumption. Therefore the number of basic infeasible variables 
strictly decreases in finitely many iterations. oO 


When, after some iteration, the number of basic infeasible variables strictly 
decreases, but does not vanish, a new auxiliary problem is constructed, whose 
objective function (infeasibility-sum) involves less variables than its predecessor, 
and iterations are carried out, until infeasibility of the original problem is detected 
or a feasible simplex tableau is reached. Involving a series of infeasibility-sums as 
objective functions in general, this method is sometimes referred to as method with 
piecewise sum of infeasibilities. 

The associated steps are put into the following algorithm. 


Algorithm 14.1.1 (Tableau Phase-I: Infeasibility-Sum) Initial: simplex tableau 
of form (3.6). This algorithm finds a feasible simplex tableau by handling auxiliary 
programs of form (14.3). 


. Stop if bj > 0,Vi = 1,...,m (a feasible simplex tableau reached). 

. For all 7 € N, compute z; = ier a,j, where I = {i =1,...,m| b; < O}. 

. Stop if zy > 0 (the original problem is infeasible). 

. Form a simplex tableau to the auxiliary problem (14.3). 

. Call one iteration of the simplex algorithm with Rule 14.1.1 being used for row 
pivoting. 

6. Go to step 1. 


nAbWN Re 


Example 14.1.1 Solve the following problem by Algorithm 14.1.1 in Phase-I: 


min f = —x,; + x2 — 2x3, 

s.t. xy — 3x2 — 2x3 + x4 = —4, 
x1 — x2 + 4x3 —x5 = 2, 
—3x1 + x2 +%x34+ x6 = 8, 
xj 20, joHl,...,6. 
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Answer Phase-I. Multiply the second equation by —1 to turn the coefficient of 
x5 to 1. Setting J = {1, 2}, construct auxiliary problem 


min f = —-X4 —X5, 

S.t. xy — 3x2 — 2x3 +4x4 = —4, 
—x, $ x2 — 4x3 +x5 = —2, 
—3x, + x2 +43+ x6 = 8, 
xj>0, j=1,2,3,6. 


Its initial tableau is 


Xx] x2 X3 X4 X5 x6 | RHS 

1 -3 —-2 1 —4 

-1 1 -4 1 —2 

—3 1 1 1 8 
-1 -l 


Add the first and the second rows to the objective row to eliminate the nonzero 
basic entries, yielding the simplex tableau below: 


xX] x2 x3 4X4 %XxX5 X» | RHS 
1 -3 —2 1 —4 
—1 1 —-4 1 —2 
—3 1 1% 1 8 
—2 -6 —6 


Iteration 1: 
q = 3; min{8/1} = 8, p = 3.So 1 in the x3 column and the 3rd row is the pivot. 
Add 2, 4, 5 times of row 3 to rows 1, 2, 4, respectively. 


Xx] xX.  X3 X4 %\X5s Xx6 | RHS 
-5 = 1 2 12 
-13 5 1 4 30 
—3 1 1 8 
—18 4 6 42 


The right-hand side of the preceding tableau is nonnegative, hence Phase-I is 
finished. To go to Phase-II, use coefficients of the original objective function to 
overwrite the bottom (objective) row. 

Add 2 times of row 3 to the bottom row to eliminate nonzero basic entries, 
obtaining feasible simplex tableau: 
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xX] x2 x3 x4 x5 x6 | RHS 
-5 -l 1 2 12 

—13 5 1 4 30 
3 1 1 1 8 
—-1 1 -2 

xX] xX. X3 XxX4 %\2Xs x6 | RHS 
= = 1 2 12 

—13 5 1 4 30 
—3 1 1 1 8 
-7 3 2, 16 


Carry out Phase-II from the preceding tableau. By the conventional rule, x, 
column is selected as the pivot column. All components of this column are 
nonpositive, as detects unboundedness of the original problem. 

In practice, used is usually the following revised version of Algorithm 14.1.1. 


Algorithm 14.1.2 (Phase-I: Infeasibility-Sum) Initial: (B, N), Bo!) xp — 
B~'b. This algorithm finds a feasible basic solution to the standard LP problem 
(12.1) by handling auxiliary programs of form (14.3). 


1. Stop if xg => 0 (producing a feasible basic solution). 
—1, Ifx;, <0, . 
ie i=l,...,m. 

0, Ifx;, = 0, 

3. Call one iteration of the simplex algorithm, in which Rule 14.1.1 is used to 
determine a pivot row index. 

4. Stop if it terminates as step 3 (or step 2) (the original problem is infeasible). 

5. Go to step 1. 


2. Construct cg:cj, = | 


14.2 Single-Artificial- Variable Algorithm 


Although the infeasible-sum method is artificial variable free, the row pivot rule 
used in it is cumbersome, compared with the normal row rule. This section describes 
a method, which uses the normal row rule and only involves a single artificial 
variable. 

Let (B, N) be basis and nonbasis of the standard problem (12.1). There are two 
schemes for the method. 

Scheme 1: The auxiliary problem is based on the original data of the problem. 

Given an m-dimensional vector h > 0 such that b— Bh ¥ 0. Introducing artificial 
variable f and normalized vector 


Gn41 = (b — Bh)/|\b — Bhi, (14.10) 
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construct the following auxiliary problem: 


min f, 


(14.11) 
s.t. Ax+ fan41 =b, x, f >0. 


It is verified that 


xp=h, xy=0, f =||b— Bh 


is a feasible solution to the problem, though not necessarily a basic solution. For 
instance, setting h = e > O seems to be helpful to avoid zero stepsize, though 
according a, 1 would lose sparsity. 

The constraint system of the auxiliary problem is equivalent to 


xp =b—Gns1 f —Nxn, (14.12) 
where 
b=Bo'b, Gna, = Bo'ans1, N= BN. (14.13) 


If b > 0, then f is clearly allowed to be zero. Thus, erasing the artificial variable 
leads to a feasible basic solution to the original problem; otherwise, the following 
applies. 


Proposition 14.2.1 If b 7 0, the row index set {i =1,...,m | Gin41 < 0,5; < 0} 
is nonempty. 


Proof We show that if b; < 0 for some i € {1,..., m}, then Gj,,41 <0. Assume it 
is not the case, i.e., 


Gin+1 = 0. 
Then from the preceding, (14.10) and (14.13), it is known that 
0 < Ging =e) Bo an41 =e) B'(b — Bh)/||b — Bh\| = b; — h;)/\|b — Bhi, 


hence b; > h; > 0, leading to a contradiction. oO 


Now assume b H 0. If xy = 0 is fixed and the value of x, decreases from 
||b — Bh||, then xg changes from h accordingly. If bj < 0 for some i € {1,..., m}, 
the according inequality constraint may block up the value of x,41 from decreasing 
to zero. In fact, the smallest value of f satisfying, inequalities 


xp=b—-—Gniif>=0, f>0, 
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ot = by fap ny = min{bj/djn+1 | ainsi < 0,6; <0,i =1,...,m} > 0. 
(14.14) 


As the value of x,+1 decreases, the according value of xj, hits O first. Let x;, leave 
and f enter the basis, then the resulting basis corresponds to a feasible basic solution 
to the auxiliary problem, with its objective value strictly decreasing. As it is lower 
bounded, solving the auxiliary problem will render its optimal solution. 

It is not difficult to show that a feasible basic solution or basis to the original 
problem can be obtained if the optimal value of the auxiliary problem is equal to 
zero (f leaves the basis). The original problem is infeasible if the optimal value of 
the auxiliary problem is positive. 

Scheme 2: The auxiliary problem is based on a canonical form of the constraint 
system. 

Assume that the canonical form, associated with basis B, is available and that 
b = B~'b # 0. Giving an/the m-dimensional vector h > 0, and defining 


Gn41 = (b—h)/||b —hll, (14.15) 


construct the following auxiliary problem: 


min f 
s.t. xg =b— fans, — Nxy, (14.16) 
x,f >=>0. 


It is clear that there is a feasible solution to the problem; that is, 
kp=h, Xy=0, f=|b-hl. 


Thus, determining pivot column index g = n+ 1 and row index p satisfying (14.14), 
one can obtain a feasible basis by performing according basis change. It is noted that 
Proposition 14.2.1 is still valid in this case. 

As for how to determine h, it is only required that h > 0, in principle. With 
respect to practice, however, it is not proper for the associated ||a,+1|| being too 
large. A simple choice is h = e > 0. Another choice is setting 


and taking 


6;, if bj < 8, . 
hj — - 2 —— 
b;, otherwise, 
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where 6; > 0,7 = 1,...,m, are a set of constants, which are equal or close to zero. 
Consequently, (14.15) becomes 


= b; — 6;, if b; < 4;, , 
Pants 0, in the other case, ; a ( ) 


We prefer the latter because it is better for maintaining sparsity of a,+1. 

Specifically, assume availability of a simplex tableau, say (3.6), of the original 
problem. Inserting the artificial f column, we obtain the auxiliary simplex tableau 
below: 


BAN FRE (14.18) 
IN an+1 b 
where the original objective row is excluded. Determine row index r by (14.14). 
Then drop xj, from and enter f to the basis by elementary transformations, so that f 
becomes the rth basic variable. Taking the rth row as the objective row, the auxiliary 
problem can be solved by the simplex method. If the optimal value vanishes and f 
has left the basis, a feasible simplex tableau to the original problem is obtained. If 
the optimal value is nonzero, the original problem is infeasible. 
The steps can be summarized into the following algorithm. 


Algorithm 14.2.1 (Tableau Phase-I: Single-Artificial- Variable) Initial: a sim- 
plex tableau of form (14.18). This algorithm finds a feasible simplex tableau to the 
standard LP problem (12.1). 


1. Select a row index r € arg min{b; |}i=1l,...,mb}. 

2. Stop if b,. >0 (infeasible problem). 

3. Determine row index r € arg min{h; /Gjn41|din+1 < 0, b; <0O,i=1,...,m}. 
4. Carry out elementary transformation to convert d,,,41 to 1 and eliminate all 


other nonzeros of the column. 
5. Determine column index g € arg maxjey G,;. 
6. Stop if a,; <0. 
7. Determine row index p € J = arg min{b; /j ¢|ai g >0; i=1,...,m}. 
8. Set p =rifr eT. 
9. Carry out elementary transformations to convert pq to | and eliminate all other 
nonzeros of the column. 
10. Stop if p = r (feasibility achieved). 
11. Go to step 5. 


It is not difficult to transform Algorithm 14.2.1 to a revised version, as is omitted. 
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Example 14.2.1 Find a feasible simplex tableau to the following problem by 
Algorithm 14.2.1: 


min f = xj + 3x2 — 2x3 + 6x4, 

s.t. —xy +2x2-—2434+4%5 =-1, 
—3x, + x2 +2x34+ x6 =2, 
x1 — 3x2 — 3x3 + x4 = —4, 
xj 20, j=l,...,6. 


Answer The constraint system is of a canonical form. Set 6; = 52 = 0. Introducing 
artificial variable x7, construct an auxiliary problem of form (14.16): 


min x7, 

s.t. —xy x2 -— 43 +45 —2x7 = —1, 
—3x, + x2 + 2x3 4x6 = 2, 
x1 — 3x2 — 3x3 4+ x4 — 447 = —4, 


6p 20, FS AysicsT. 


In order to turn to Phase-II conveniently after the Phase-I is finished, we put the 
objective row of the original problem as the bottom row. The initial tableau is 


Iteration 1: 

1. min{—1, 2, -—4} = —4 < 0. 

3. max{—1/— 1, -4/-4}=1,r=1. 

4. Multiply row 1 by —1, then add 4 times of row | to row 3: 


5. max{l,—-1,1,-1}=1>0, qg=1. 
7. min{1/1,0/5} = 0, p =3. 
8. p#l. 
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9. Multiply row 3 by 1/5, then add —1,3,—1 times of row 3 to rows 1, 2, 4, 
respectively: 


XxX] x2 x3 X4 X5 X6 
2/5 4/5 -1/5 —1/5 

-16/5 13/5. 3/5. 12/5. 1 
1-7/5 1/5* 1/5 4/5 
22/5 11/5 29/5 4/5 


Iteration 2: 


5. max{2/5, 4/5, —1/5, 1/5} = 4/5 > 0, q =3. 

. min{1/(4/5), 2/(13/5), 0/(1/5)} = 0, p = 3. 

.p#ét. 

. Multiply row 3 by 5, then add —4/5, —13/5, 11/5 times of row 3 to rows 1, 2, 4, 
respectively: 


\o on 


x1 x2 X3 X4 x5 X6 
—4 6 -1 3 

—13 15* —2 8 1 
5 ed gs) 1 —-4 
i. 11 8 -8 


Iteration 3: 


5. max{—4, 6, —-1,3} =6>0, q =2. 

. min{1/6, 2/15} = 2/15, p =2. 

.p#étl. 

. Multiply row 2 by 1/15, then add —6,7, 11 times of row 2 to rows 1, 3, 4, 
respectively: 


\o on 


xX x2 X3 X4 X5 X6 
6/5* -1/5 -1/5  —2/5 
—13/15 1 —2/15 8/15 1/15 
—16/15 1 1/15 —4/15 7/15 
22/15 95/15 —32/15 11/15 


Iteration 4: 

5. max{5/6, —1/5, —1/5, —2/5} =5/6>0,q =1. 
7. min{(1/5)/(11/5)} = 1/11, p= 1. 

8. p=l. 
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9. Multiply row | by 5/6, then add 13/15; 16/15; —6/5; —22/15 times of row 1 to 
rows 2, 3, 4, respectively: 


xX) x2 x3 X4 X5 X6 X7 RHS 
1 —1/6 -1/6 —-1/3 5/6 1/6 
1 —5/18 7/18  —2/9 13/18 | 5/18 

1 -1/9 4/9 1/9 8/9 | 10/9 


61/9 —17/9 11/9 —11/9 | 11/9 


As the artificial variable x7 has already left the basis, the resulting is a feasible 
simplex tableau to the original problem. 

It is noted that 5; and 52 above were set to 0, and hence no progress was made in 
the first 3 iterations due to degeneracy. Although parameters 6;,i = 1,...,m, are 
required to be nonnegative theoretically, therefore, they should be set to different 
positive values practically. Moreover, it seems appropriate to set 6; < 1 so that 
||@n+1|| is not too large. 


14.3. Perturbation of Reduced Cost 


As a result of not involving reduced costs, Algorithm 14.4.2 could generate a 
feasible solution that is far from dual feasible, and hence the number of iterations 
required subsequently by Phase-II would be quite large. In this aspect, the Phase-I 
method, presented in this section, seems to be advantageous (Pan 2000). 

It solves an auxiliary perturbed problem. Let (3.6) be an infeasible simplex 
tableau, and let 6; = 0, j € N, be predetermined perturbation parameters. The 
auxiliary problem results from perturbing reduced costs to nonnegative values. More 
precisely, introduce the index set 


J={jeN|zj <5}, JHN\V. (14.19) 


Replacing reduced costs z; in the simplex tableau by 
z= ee (14.20) 
zj,ifje N/J 
leads to a dual feasible tableau, associated with a perturbed problem. As a result, 


the dual simplex algorithm can get itself started, until optimality of the perturbed 
problem is achieved, or dual unboundedness of it is detected. 


Theorem 14.3.1 [f the auxiliary perturbed problem is dual unbounded, the original 
problem is infeasible. 
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Proof The perturbed problem has the same constraints as the original problem. 
Unboundedness of the former implies that there is no feasible solution to the former, 
and hence so to the latter. Oo 


Assume that optimality of the perturbed problem is achieved. We might as well 
assume that (3.6) is again the final tableau, associated with basis B. Its right-hand 
side is now non-negative. Consequently, computing 2y = cy —N' B~'cg (restoring 
reduced costs of the original problem) and covering Zy with Zy lead to a feasible 
simplex tableau to the original problem. 

The overall steps are put into the following algorithm. 


Algorithm 14.3.1 (Tableau Phase-I: Reduced Cost Perturbation) Assume we 
are given the perturbation parameters 6; > 0, j € N. Initial: simplex tableau 
of form (3.6). This algorithm finds a feasible simplex tableau to the standard LP 
problem. 


1. Perturb: z; = 6;,Vj €{j7 € N |Z; < 4;}. 

2. Call the dual simplex algorithm. 

3. Stop if it terminates at step 3 (infeasible problem). 

4. If it terminates at step 2, compute zy = cy — N™B-!cp. 
5. Stop (feasibility achieved). 


Note There exists a perturbation-free variant of the preceding algorithm: in each 
iteration, one uses 7, determined by (14.20) in place of Zy for the minimum-ratio 
test, therefore saving computations for restoring reduced costs. 


Algorithm 14.3.1 performed very well in preliminary computational experiments 
(Pan, 2000). A reason seems to be that not all perturbed reduced costs affect the 
final outcome. Sometimes, reduced costs restored in step 4 of Algorithm 14.3.1 are 
themselves nonnegative, and hence there is no need for carrying out Phase-II. In 
fact, it is verified that the manipulations in step 2 amount to solving the following 
problem: 


min bly, 
st. ATy <é, 


where 


‘ alB-"cg +-4j, ifj € J, 
TR ice 
Zj, ifj ¢ N/J. 
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That is to say, perturbing z; < 6; to z; = 6; amounts to slackening dual constraint 
ay y<c; to 


ajy < a; B"cp +6) =cj + Oj —cj + a; B-'cp) =cj + (6; —Z;). 


If the slackened constraints are all inactive for dual optimal solutions, or in 
other words, nonnegative constraints of primal variables, associated with perturbed 
reduced costs, are active for primal optimal solutions, the primal and dual optimal 
solutions will not be affected by such perturbations at all (for detailed analysis, see 
Pan, 2000). 


Example 14.3.1 Solve the following problem using Algorithm 14.3.1 in Phase-I: 


min f = xj + x2 — 3x3, 

s.t. —2xy — x29 +4x34+ x5 = —4, 
xy — 2x2 + x34+%6=5, 
—x1 + 2x3 4+ x4 = —3, 
xj 20, j=Hl,...,6. 


Answer 

Phase-I. 

Initial: There is an available simplex tableau (B = {5, 6, 4}) to this problem. 
Perturb negative reduced cost —3 to 0, so that it becomes dual feasible. As an 
illustration for implementation, we add the perturbed reduced costs at the bottom 
(63 = 0). 


xX] x2 x3 X44 + %XxX5 X6 | RHS 
—2* —1 4 1 —4 
1 -2 1 1 5 
—1 2 1 =3 
1 1 -3 
1 1 


Taking the bottom row as the objective row, execute the dual simplex algorithm. 
Note that the second (objective) bottom row does not take a part in pivoting. 
Iteration 1: 


1. min{—4, 5, —3} = -4 <0, p=1. 

3. J = {LF O. 

4, min{—1/(—2), -1/(-D} = 1/2, gq =1. 

5. Multiply row | by —1/2, then add —1; 1; —1; —1 times of row 1| to rows 
2, 3,4, 5, respectively: 
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x4 x2 x3°«OX4 X5 
1 ve ae 1/2 
=$/2- - 3 1/2 
1/2 1 -1/2* 
12-754 1/2 
io 3 1/2 


Iteration 2: 


. min{2,3,-1} = -1 <0, p=3. 

JS = {5} AV. 

- minf{—(1/2)/(-1/2)} = 1, q =5. 

- Multiply row 3 by —2, then add 1/2, —1/2, —1/2, —1/2 times of row 3 to rows 
1, 2, 4, 5, respectively: 


an BR We 


1 =2 = 
a si 1 

-1 —2 1 
LS 1 
1 2; 1 


Optimality of the perturbed tableau is achieved. Deleting the perturbed (bottom) 
row gives a feasible simplex tableau of the original problem. 

Phase-II. 
Iteration 3: call the simplex algorithm. 

Select x3 column as the pivot column and the 2nd row as the pivot row. Multiply 
row 2 by 1/3, then add 2; 1; —2 times of row 2 to rows 1, 4, 5, respectively: 


xX] x2 x3 x4 x5 
1 4/3 -1/3 
2/3. 4 1/3 

=f. = | 
1/3 4/3 


The reduced costs are all nonnegative, optimality is hence achieved. The optimal 
basic solution and optimal value: 


x* = (13/3, 0,2/3,0,2,0)',  f* =7/3. 


Although the perturbation parameter was set to 0 in the preceding example, 
the positive parameter value should be practically favorable for the sake of anti- 
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degeneracy. It seems to be suitable to use 
3j >e>QO0, 


where € is the feasibility tolerance (€ = 107°, or so). Computational experiences 
indicate that the perturbation method is not sensitive to the magnitude of the 
parameters although exceeding 10~! seems to be inadvisable. 


14.4 Using Most-Obtuse-Angle Column Rule 


It is somehow surprising that the dual pivot rule can be used to achieve primal 
feasibility and perform well in preliminary computational experiments (Pan 1994). 
That is to say, whether the reduced costs are all nonnegative or not, the classical 
dual pivot rule can be applied. The Phase-I method presented in this section is even 
simpler, as it involves neither the minimum-ratio test nor reduced costs. 

Assume that a simplex tableau, say (3.6), is infeasible, and a pivot row index p 
has been selected by the conventional rule 


p € argmin{b; |i = 1,..., m}. (14.21) 


Therefore, it holds that 5 p < 0. 
The key of the presented Phase-I method is to use the following column rule. 


Rule 14.4.1 (Most-Obtuse-Angle Column Rule) Determine the pivot column 
index 


€ arg min dp j. 14.22 
q € arg min dp ; (14.22) 


In case of dpg < O, after the basis change, the basic infeasible variable Xjp 
becomes nonbasic (hence feasible), and the pth component of the right-hand side of 
the tableau changes from negative to positive, that is, b p= b p/Apq > 9. 

The associated steps can be summarized into the following algorithm. 


Algorithm 14.4.1 (Tableau Phase-I: The Most-Obtuse-Angle Column Rule) 
Initial: simplex tableau of form (3.6). This algorithm finds a feasible simplex tableau 
to the standard LP problem (12.1). 


. Select row index p € arg min{b; |i=1,...,mb}. 

. Stop if by = 0. - 

. Determine column index q € arg Min jen Ap j- 

. Stop ifap,q = 9. - 

. Carry out elementary transformations to convert dp, q to | and eliminate all other 
nonzeros of the column. 

6. Go to step 1. 


MARWNOH 
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It is noted that only the right-hand side and the pivot row of the simplex tableau 
are used in each iteration of the preceding Algorithm. 


Theorem 14.4.1 Assume termination of Algorithm 14.4.1. It terminates either at 


(i) Step 2, obtaining a feasible basic solution or at 
(ii) Step 4, detecting the infeasibility of the problem 


Proof It is clear that if it terminates at step 2, then b > 0, and hence a feasible 
simplex tableau is achieved. In case of the termination at step 4, it holds that b p <0 
and that a, ; => 0, Vj € N. According to Theorem 3.3.1 of Part I, there is no feasible 
solution to the problem. Oo 


The algorithm is not a monotone one, that is, the objective value would not 
monotonically change in the solution process. Its finiteness is hence not guaranteed 
even if nondegeneracy is assumed. Although Guerrero-Garcia and Santos-Palomo 
(2005) offer a cycling instance for this algorithm, it may be still expected that 
cycling hardly happens in practice. 

The geometric meaning of the most-obtuse-angle column rule can be revealed 
from the dual space. Let (y, z) be the current dual basic solution. The (m + n)- 
dimensional vector 


a —BT 
d=| oy | =| N'B" Je, (14.23) 
OB I 


is an ascent direction. In fact, it is verified that d is in the null space of coefficient 
matrix of the dual constraint equalities, satisfying 


A'd+o=0, 
and forms an acute angle with the dual objective gradient, i.e., 
(b', 0, 0)d = b'h = —xj, > 0. 


The negative og implies that d forms the maximum possible obtuse angle with 
the gradient é+ of the dual nonnegative constraint z; > 0. Therefore, if the 
ascent direction is close to the dual objective gradient, then e+ tends to form 
the most obtuse angle with the dual objective gradient. According to the heuristic 
characteristic of optimal basis (Sect.2.7 of Part I), it is favorable to let the dual 
constraint z, > 0 be satisfied as equality, accordingly entering xq to the basis. 

We stress that the use of the conventional dual rule in step 1 is just for simplicity, 
which does not mean it is the best choice. In fact, it should be much better to use 
the rules presented in Chap. 13 instead. From a geometric point of view, the most- 
obtuse-angle column rule should be best matched by the dual steepest-edge rule 
Il. 
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Nevertheless, Rule 14.4.1 does not employ information associated with the 
objective function at all. Considering the extent to which the current vertex violates 
dual constraints, we suggest the following variant. 


Rule 14.4.2 (Variant of Most-Obtuse-Angle Column Rule) Assume we are given 
a constant t such that 0 < t < 1. Select pivot column index 


q € argmin{zZ; | dp); < tO, j © N}, gn pj <0. 
JE 


In order to widen the range of choice, t should be close to 1, so that in the case 
when Zy has negative components, the dual constraint inequality, that is violated the 
most, will be satisfied as equality. It might be suitable to set o = 0.95, or so. 

The Phase-I algorithm seems very promising, not to mention its dual algorithm, 
which performs very well in solving large and sparse problems in computational 
experiments (Sect. 15.4). It is the author’s favorite Phase-I algorithm not only 
because of its simplicity and good numerical stability but also because of its 
geometric meaning based on the most-obtuse-angle heuristic of the optimal solution. 


Example 14.4.1 Find a feasible simplex tableau to the following problem by 
Algorithm 14.4.1: 


min f = 2x1 = N95 

s.t. 2x, —x2+ x3 = —2, 
x} + 2x2 + x4 = 3, 
—8x, + x2 —x4 + x5 = —4, 
xj 20, jgol,...,5. 


Answer Initial: adding the second equality constraint to the third to eliminate x4. 
This results in the following simplex tableau: 


x] x2 x3 X4 x5 | RHS 
2 —I1* 1 —2 
1 2 1 3 

-7 3 1 -1 
2 —1 


Iteration 1: 


1. min{f—2,3,-—-1} = -—2 <0, p=1. 

3. min{2, -1} = -1 <0,g =2. 

5. Multiply row 1 by —1, then add —2, —3,1 times of row | to rows 2, 3, 4, 
respectively: 
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x1 x2 X3 x4 x5 | RHS 
—2 1 -1 2 
5 2 1 -1 
—1* 3 1 —7 
4 2 


Iteration 2: 


1. min{2,—1, —7} = -—7 <0, p=3. 
3. min{f—1,3} = —-1 <0, g=1. 
5. Add 2, 5 times of row 3 to rows 1, 2, respectively: 


x1 x2 X3 X4 X5 RHS 


1 -7 16 

17 1 34 

1 =o —1 7 
—1 2 


The right-hand side of the above tableau is nonnegative, hence the obtained is a 
feasible simplex tableau, from which Phase-II can get itself started. 

In practice, usually used is the following revised version of the preceding 
algorithm. 


Algorithm 14.4.2 (Phase-I: Most-Obtuse-Angle Column Rule) Initial: (B, NV), 
B-!, xp = B7'b. This algorithm finds a feasible basic solution to the standard LP 
problem (12.1). 


Select pivot row index p € arg min{x;, |i = 1,..., mb}. 
Stop if x; = 0 (feasibility achieved). 

Compute h = B'e,, on = Nth. 

Determine pivot column index g €= arg minjey aj. 
Stop if og = 0 (infeasible problem). 

Compute ag = Bla, anda = xj,/0q. 

Set X, = a, and update:xz = xg — ad, ifa £0. 
Update Bo!. 

Update (B, N) by exchanging j, and q. 

Go to step 1. 


SOOO NON ae aa 
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Chapter 15 ® 
Dual Simplex Phase-l Method crests 


The mission of a dual Phase-I procedure is to provide an initial dual feasible basis 
to get the dual simplex method started. 

In the dual simplex context, one can establish a dual Phase-I by introducing n—m 
dual artificial variables, with or without penalty terms in the objective function, just 
as in the primal context. We will not touch these methods because they are not 
suitable for applications (interested readers are referred to, e.g., Padberg 1995). 

This chapter is devoted to practicable dual Phase-I methods, which can be 
regarded as dual counterparts of the simplex Phase-I methods. 


15.1 Dual Infeasibility-Sum Algorithm 


This section will present a dual version of the infeasibility-sum method. Appearing 
in the early days, the basic idea of the method draws new attention and is put 
into effect in practice in recent years because of its good performance (see, e.g., 
Koberstain and Suhl, 2007). 

Assume that dual basic solution 


y=B'cg; in=cy—N'y, Ze=0 (15.1) 
is infeasible, i.e., zy # O. Introduce the index set 


J={jeN|zj <9}. (15.2) 
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Taking the second term of the sum of the dual infeasible variables, i.e., 


T 


=e -(>_ 4) 2 


Jed jed jel 
as objective function, we construct the following auxiliary problem 


max = 9 aj)"y, 
s.t. Bly + 7B =CB, 
Nly+zn =cn, 
Zp = 0; zjz0, JEeN\, 


(15.3) 


whose nonnegativity constraints only impose on the variables corresponding to the 
nonnegative components of the dual basic solution. 

Clearly, there is a basic solution to the primal problem of the auxiliary problem; 
Le., 


ta~B > ay. ays, (15.4) 
jet 
where xz it termed auxiliary right-hand side. 


Theorem 15.1.1 (Dual Infeasibility Test) Jf xg > 0, the original problem is 
infeasible or unbounded. 


Proof Assume that y’ is a feasible solution to the original dual problem such that 
z’=cy —N'y'>0. 
Thus, it holds that 
/ T / 
= he-(o4) y'>0. (15.5) 
jed jet jeJ 


On the other hand, it is known from the assumption that (15.4) gives a feasible 
basic solution x to the primal problem of (15.3), and (15.1) gives a complementary 
feasible solution (y, z) of (15.3, hence the two are a pair of optimal solutions. It is 
clear that (y’, z’) is a feasible solution to (15.3), associated with objective value no 
more than the optimal value, hence 


yz = pays < 0, 


jeJ Jed 
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where the last inequality comes from (15.2). This contradicts (15.5). Therefore, 
there is no feasible solution to the original dual problem when xg > 0. According to 
the duality principle, the original (primal) problem is infeasible or unbounded. O 


In case when xg # O, carry out one dual simplex iteration with a modified 
column pivot rule. To this end, assume that row index p has been determined such 
that 


Sip HO. (15.6) 
Consider the following scheme for updating dual solution: 
J=¥—-BB e,, Z=Z+Hhv, v=A'B ey, (15.7) 


where stepsize 6 is defined by the rule below: 


Rule 15.1.1 (Auxiliary Column Rule) Select pivot column index q such that 


min{—Z;/v; | vj <0, j e J} if{i e N\J | vj <0} FG, 
max{—Zj;/v; | vj > 0,7 € J} otherwise. 


p= —Zq/Vq = 
(15.8) 


The following results are counterparts of Lemma 14.1.2 and Theorem 14.1.3 in 
Sect. 14.1, proofs of which are therefore omitted. 


Lemma 15.1.2 Let p be a row index, satisfying (15.6). Then (15.8) is well-defined, 
and the expression (15.7) gives a new feasible basic to the auxiliary problem. 


Theorem 15.1.3. Assume that the original dual problem is feasible. If the dual 
feasible variables are all nondegenerate, the number of dual infeasible variables 
strictly decreases in finitely many iterations. 


If the number of infeasible variables reduces but does not vanish after an 
iteration, the process is carried out with an updated auxiliary dual problem, until 
dual feasibility is attained or dual infeasibility or unboundedness of the original 
problem is detected. 

The steps are put in the following algorithm. 


Algorithm 15.1.1 (Dual Phase-I: Dual Infeasibility-Sum) Initial: (B, NV), Bo}, 
y, z. This algorithm finds a dual feasible solution by handling auxiliary programs of 
form (15.3). 


. If 7 > 0, compute xg = B~'d, and stop (dual feasibility achieved). 

. Compute +g = —B™! Dies Ui: where J = {j € N |Z; < O}. 

. Stop if xg > 0 (the original dual problem is infeasible or unbounded). 

. Call one iteration of the dual simplex algorithm with the Rule 15.1.1 used for 
column pivoting. 

5. Go to step 1. 


RWNe 
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Example 15.1.1 Find a dual feasible basic solution of the following problem by 
Algorithm 15.1.1: 


min f = —x; + x2 — 2x3, 

S.t. xy — 3x2 — 2x3 + x4 = —4, 
x1 + x2 — 4x3 +%x5 = —2, 
—3x1 + x2 +x34+ x6 = 8, 
oO, JH haiwh 


Answer Taking J = {1,3}, construct an auxiliary right-hand side xg = 
—(-1, —5, —2)!, Since components of it are all nonnegative, it is asserted that 
the problem is infeasible or unbounded. In fact, the problem is just the same as that 
in Exam. 14.1.1, where the infeasibility of the problem is declared in Phase-II. 


Example 15.1.2 Find a dual feasible basic solution to the following problem by 
Algorithm 15.1.1: 


min f = —5x; — 7x2 + x4, 

s.t. xy + 2x2 + x3 =3, 
2x, + x2 — x4 +X5 = —2, 
—xy + x2 +4x4+2x6= —-1, 
xj 20, jol,...,6. 


Answer Taking J = {1,2}andN\J = {3,4,5,6}, construct the auxiliary 
problem. B = {3,5,6},N = {1,2,4},B-! = I,zy = (—5,—7,1)". The 
auxiliary right-hand side is xg = (—3, —3, o)?. 

Iteration 1: Carry out a single iteration of the revised dual algorithm; in which (15.8) 
is used for column pivoting instead. 


1. min{—3, —3, 0} = —3, p = 1, so x3 leaves the basis. 
a: on= N's le, = (1,2,0)". 
5. B = max{—z;/oj|o; < 0,7 = 1,2,3} = max{5/1,7/2} = 5,¢g = 1, so x1 
enters the basis. 
6. Zy = Zn + Bon = (—5, —7, 1)' +5 x (1, 2,0)? = (0,3, 1)", 2), =5. 
. Gq = B-lag = (1,2, -1)'. 
8. @ = *j,/og = —3/1 = -3, 
kp = Xp—ad, = (—3, -3,0)'—-(—-3)x (1, 2, -1)? = (0,3, -3)7, #1) = 


~ 


10. B = {1,5,6}, N = {3,2,4}, zy = (5,3,1)',J = G,xp = Bo'b = 
(3, —8, 2)7, Dual feasibility attained. 


15.2 Dual Single-Artificial- Variable Algorithm 389 


Algorithm 15.1.1 is easily to converted to its tableau version. Note only that pth 
row of the simplex tableau is v' = eT B—'A, x; column is equal to Bo1a,, and in 
step 2 the following formula is used, instead: 


fgp=—) aj, J= {je Nz <0}. 
Jed 
We illustrate with the preceding example. 


Example 15.1.3 Find a dual feasible simplex tableau for Example 15.1.2 by the 
tableau version of Algorithm 15.1.1. 


Answer Initial: Clearly, there is a simplex tableau to the problem; i.e., 


x1 x2 x3 x4 xs x6 | RHS 
1* 2-1 3 

2 1 -1 1 —2 
—l 1 1 1 -1 
-5 -7 1 


Iteration 1: 

J = {1,2}, J = {4}, Xe = -((1, 2, -D™+ (2, 1, DT) = (-3, -3, 0)7 

p = 1, B = max{—(—5)/1, -(-7)/2}=5, q=1. 

Respectively add —2, 1, 5 times of row | to rows 2,3,4, resulting in the following 
tableau: 


x] x2 X3 x4 Xx5 x6 | RHS 
1* 2 1 3 
—3 —2 -1 1 —8 

3 1 1 1 2 

3 5 1 15 


which is dual feasible. 


15.2 Dual Single-Artificial- Variable Algorithm 


In this section, the idea of the single-artificial-variable method, described in 
Sect. 14.2, is applied to the dual problem to derive a dual Phase-I method. To this 
end, we introduce an artificial variable to the dual problem. The derived method has 
been used as dual Phase-1 for the so-called dual projective pivot algorithm (Pan 
2005). 
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As in the primal context, there are two schemes: 

Scheme 1: The auxiliary dual problem is based on the original data of the 
problem. 

Given any vector v > 0. Introducing the artificial variable y,+1, we construct 
the following auxiliary dual problem from constraints of the dual problem: 


Max Yn+1, 
15.9 
st. ATy+(v—c)ymy1 +z =C, z2=0 ae 


which has the feasible solution 


or 


Theorem 15.2.1 There is an optimal solution, associated with a nonpositive 
objective value, to the auxiliary dual problem (15.9). If the optimal value is equal to 
zero, then the (y, z) part of its optimal basic solution is feasible basic to the original 
dual problem; otherwise, the latter is infeasible. 


Proof To the auxiliary dual problem (15.9), there is a feasible solution with 
objective value bounded above by 0, hence there is an optimal solution with 
nonpositive optimal value. Let (y’, y/, we 2 2, 41) be its optimal solution, where 
Zp» Zy = 0. Assume that the optimal value is equal to zero, i.e., ad = 0. Since 
the optimal solution satisfies equality constraints of (15.9), it holds that 


Bt ra CB 
Nt y+ zy = {cn 
/ 
0 <n+1 0 


Thus, (y’, z’) is a feasible basic solution to the original dual problem. 

Assume now that the optimal value is less than 0, i.e., y/, 4, < O. If the original 
dual problem has a feasible solution (y”, z”), then (y”, z”) together with Veet = 
z41 = Ois a feasible solution to (15.9), therefore leading to y,,., = Yn4, = 0, 
which contradict y/, 41 < 9. Therefore, the original dual problem is infeasible if the 


optimal value of (15.9) is less than 0. oO 


According to the preceding Theorem, if the dual simplex method is used to solve 
the auxiliary dual problem, a dual feasible basic solution to the original problem can 
be obtained, if any. 

Scheme 2: The auxiliary dual problem is based on a canonical form of the 
constraint system. 

Let (B, N) be basis and nonbasis of the standard LP problem, associated with a 
simplex tableau, say (3.6). Given any Zy > O. Introducing artificial variable x,,4+1, 
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construct the following auxiliary problem: 
min f = f+zh xy, 


= XB 
I N 0 0 (15.10) 
xe = ; > 0, > 1, 
’ Ge, a) ap Ge ea 


where 
N=B"!N, Zw =cn — N'B cg, f =b'B lcs. 


Clearly, (x = 0, X,4+1 = 1) is asolution to the preceding problem. We are interested 
in its dual problem; i.e., 


max w = re? Ym+1; 
ZB 0 I 0 
s.t. zy |= | zy | —| NT in —Zn ( 4 i Z,Znt1 = 0. 
Zn+1 0 0 1 ant 
(15.11) 


The above is just the auxiliary dual problem we want, as it is much simpler than 
(15.9), and has a feasible solution 


A 0 ZR 0 
ie Veal ). gy |=| an 1 20. (15.12) 
Ym+1 —1 re 
Zn4+1 1 


Like Theorem 15.2.1, it can be shown that if (15.11) is solved with a zero optimal 
value, a dual feasible basic solution to the original problem is obtained; otherwise, 
there is no dual feasible solution, or the original problem is infeasible or unbounded. 
From the last equality constraint, zn41 = —ym+1, of (15.11), it is known that 
max y+) and min Z,+1 are equal, if any. 

Like the dual simplex method, it is also possible to solve (15.11) via the simplex 
tableau. In fact, problem (15.10) has the following simplex tableau, which is not 
dual feasible: 
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Adding the (m + 1)th row as the bottom row gives 


XB XN Xn41 f | RHS 
I N 
(yu —Zn)' 1 1 
ae 2Zn41 -L—f +1 


(15.13) 


where 2,41 = 1. The preceding tableau, corresponding to the dual feasible solution 
defined by (15.12), can be taken as an auxiliary initial tableau though not a simplex 
one. To turn it to a simplex one, one more basic variable is required. To this end, 
determine a dual stepsize 


B = 2q/(@q — %q) = min{zZ;/(Zj —Zj) | 2; —Z; > 0, j € N}. (15.14) 


If B > 1, set 6 = 1, g = n+ 1. Converting the entry in the (m + 1)th row and 
q column to 1, and eliminating the other nonzeros in the column by elementary 
transformations, so that g-indexed column becomes basic. If it happens that gq = 
n+ 1, then deleting the (m + 1)th row and x,4; column, and putting b in the RHS 
column gives a dual feasible simplex tableau to the original problem. In a general 
case of g #n-+ 1, obtained is only a dual feasible tableau to the auxiliary problem, 
which can be solved by the dual simple method. 

Theoretically, any nonnegative vector Zy can be used to construct an auxiliary 
problem. A simple way is to set Zy = e. But another way would be more suitable 
for sparse computations: for given 6; > 0, j € N. Setting 

» _ | Oj, if Zj < 4;, 
45 = ie otherwise, =f 


The associated steps can be summarized into the following algorithm. 


Algorithm 15.2.1 (Tableau Dual Phase-I: Single-Artificial-Variable) Initial: 
auxiliary tableau of form (15.13). This algorithm finds a dual feasible tableau to the 
standard LP problem. 


1. Determine f and q such that 
B = 2g/(q — Zq) = min{2;/(2@; — Zj;) | 2; —Z; > 0, j € N}. 


2. If B>1,setB=1,qHnt+l. 

3. Convert the entry in the (m + 1)th row and g column to 1, and eliminate all other 
nonzeros in the column by elementary transformations. 

4. Ifg =n+1, delete the (m+ 1)th row and x,41 column, and stop (dual feasibility 
achieved). 

5. Call the dual simplex algorithm (the (7 + 1)-indexed column is taken as nonbasic 
at the beginning). 
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Note In the initial tableau, the first m + 1 components of the x,+4; column and 
the RHS column are equal. When carrying out the preceding algorithm, the (n + 
1)-indexed column may be also deemed as the auxiliary right-hand side, and the 
right-hand side b of the original problem is filled in RHS column (but does not take 
a part in pivoting). In this way, once x,+, enters the basis, eliminating the (m + 1)th 
row and (n+ 1)-indexed column gives a dual feasible tableau to the original problem. 
The f column can be omitted, as it does not change at all in the solution process. 


It is not difficult to transfer Algorithm 15.2.1 to its revised version, which is 
omitted here. 


Example 15.2.1 Solve the following problem using Algorithm 15.2.1 as dual 
Phase-I: 


min f = 5x, — 2x2 —x3+X4, 
s.t. 2x7 —x2 +x34+2x4=5, 
—5x1 + 3x2 + 2x3 +%x5 = —2, 
—x1 + 2x2 -x3+xX6=—-1, 
xj 20, joHl,...,6 
Answer From the first constraint equality, it follows that 
x4 = 5— 2x, + x2 — x3. 


Substitute it to the objective function to eliminate x4: 


Ff =54+3x1 — x2 — 2x3. 


Setting 2) = 0, Z2 = 23 = 1, construct the following auxiliary initial simplex 
tableau: 
Pal x2 x3 M4 oX§ XG XT RHS 
2 -1 1 1 5 
—5 3 2 1 —2 
—l 2 —-l 1 -l 
—3 141 1+2* 1 1 
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Phase-I: call Algorithm 15.2.1.m + 1 = 4. 

Iteration 1: 

1. 6B=min{l/d+1),1/d4+2),} =1/3 < 1,¢ =3. 

3. Multiply row 4 by 1/3, then add —1, —2, 1, —1 times of row 4 to rows 1, 2, 3, 5, 
respectively: 


xX] x2 X33 X44 X5 XG RHS 
3. —5/3 1 14/3 
—3* 5/3 1 —8/3 
—2 8/3 1 —2/3 
-1 2/3 1 1/3 
1 1/3 —13/3 


5. Call the dual simplex algorithm (tableau form). Note that x7 column also 
represents the auxiliary right-hand side. 


Iteration 2: 


1. min{—1/3, —2/3, 1/3, 1/3} = —2/3 < 0, p=2. 

4. min{—1/(—3), (—2/3)/(—2/3)} = 1/3, q =1. 

5. Multiply row 2 by —1/3, then add —3, 2, 1, —1 times of row 2 to rows 1, 3, 4, 5, 
respectively: 


xX) x2 X3 X4 X5 X6 X7 
1 1 —l* 

1 5/9 -1/3 2/9 
14/9 2/3 1 7/9 
9 1 =1/3 5/9 
8/9 1/3 4/9 


Iteration 3: 


1. min{—1, 2/9, 7/9, 5/9} = —1 <0, p=1. 

4. min{—(4/9)/(—l)} = 4/9, ¢ =7. 

5. Multiply row 1 by —1, then add —2/9, —7/9, —5/9, —4/9 times of row | to rows 
2, 3,4, 5, respectively: 
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nal x2 x3 X4 x5 X6 7 RHS 
-1 -1 1 —2 

1 —5/9 2/9 —1/9 4/3 

14/9 7/9 1/9 1 8/3 

1/9 1 5/9 2/9 7/3 

8/9 4/9 7/9 —13/3 


Optimality of the auxiliary problem is achieved with the artificial variable x7 
entering the basis. Deleting the first row and the seventh column leads to a dual 
feasible simplex tableau to the original problem; i.e., 


nal x2 x3 X4 XS X6 RHS 
1 —5/9 2/9 —1/9 4/3 
14/9 79 1/9 1 8/3 
19 1 5/9 2/9 7/3 
8/9 4/9 7/9 13/5 


min{4/3, 8/3, 7/3} => 0. Optimality is achieved, hence there is no need for dual 
Phase-II. The optimal basic solution and objective value are 


x* = (4/3, 0,7/3,0,0,8/3)), f* = 13/3. 


15.3. Perturbation of the Right-Hand Side 


The basic idea of this method is simple. It perturbs the right-hand side of the simplex 
tableau to nonnegative and converts the resulting tableau to an optimal one via the 
simplex method; then, it recomputes the right-hand side from the original data to 
obtain a dual feasible simplex tableau. 

Let (3.6) be an initial simplex tableau and let the perturbation parameter be given 
by 


To weaken effect of degeneracy, we recommend that 4; take positive values, different 
from each other. The method replaces b by 


te (15.15) 


: b;, otherwise, 


leading to a feasible simplex tableau. 
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We claim the following without proof. 


Theorem 15.3.1 /f the perturbed problem is unbounded, the original problem is 
infeasible or unbounded. 


We might as well denote again by (3.6) the final tableau of the perturbed problem, 
associated with basis B. Assume that the final tableau is optimal, 1.e., z; > 0, j € 
N. To restore a simplex tableau to the original problem, compute b = B~'b and 
f=-c 2D to overwrite the old b and f, resulting in 


which is clearly a dual feasible tableau to the original problem. 
The overall steps are put into the following algorithm. 


Algorithm 15.3.1 (Tableau Dual Phase I: Right-Hand Side Perturbation) 
Given perturbation parameter 6; > 0,7 = 1,...,m, a simplex tableau of form 
(3.6). The algorithm finds a dual feasible simplex tableau to the standard LP 
problem. 


1. Perturbation: b; = 5;, Vi € {i = 1,...,m | bj < 4;}. 

2. Call the simplex algorithm. 

3. Stop if termination occurs at step 3 eee or unbounded). 

4. Cover b, f byb = B-'b, f =—c Lb if termination occurs at step 2. 
5. Stop (dual ee achieved). 


Note A variant of this algorithm results from not perturbing the right-hand side, 
but directly using b’, defined by (15.15), in place of b for the minimum-ratio test, as 
therefore saves the restoration. 


Intuitively, the perturbation of the right-hand side amounts to relaxing nonnega- 
tivity restrictions on basic variables. If these constraints are inactive at the optimal 
solution, then the perturbation is independent of the determination of the optimal 
solution. In some cases, an LP problem can be solved by Algorithm 15.3.1 alone. In 
fact, we can claim the following (Pan, 1999). 


Theorem 15.3.2 Assume the dual nondegeneracy of optimal solutions. If the 
perturbed components of the right-hand side correspond to optimal basic variables, 
Algorithm 15.3.1 generates an optimal simplex tableau. 


Example 15.3.1 Find a dual feasible tableau by Algorithm 15.3.1: 


min f = —5x, — 7x2 + x4, 

s.t. xy + 2x2 + x3 = 3, 
2x, + x2 —xX44+ x5 = —2, 
—xy + x2 +4x44+%6= —-1, 
xj>0, jHl,...,6. 
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Answer Initial: A simplex tableau can be obtained from the problem. Take 62 = 
1/6, 63 = 1/12, and put the perturbed right-hand side at the right end of the tableau. 


2 1 -1 1 
1 1* 1 1 
Fn ty 1 


Call the simplex algorithm. 
Iteration |: 


1. minf—5, —7, 1} = —7 <0, q =2. 

3. 1={1,2,3} 49%. 

4. min{3/2, (1/6)/1, (/12)/1} = 1/12, p =3. 

5. Add —2, —1, 7 times of row 3 to rows 1, 2, 4, respectively: 


x1 X20 0X30 XQ XS 
3 1 —2 

3* —2 1 
—1 1 1 
—12 8 


Iteration 2: 


. minf—12, 8,7} = -12 <0, g=1. 

-T={1, 2349. 

. min{(17/6)/3, (1/12)/3} = 1/36, p = 2. 

. Multiply row 2 by 1/3, then add —3,1,12 times of row 2 to rows 1, 3, 4, 
respectively: 


nA BWR 


xy x2 x3 x4 x5 X6 
1 -1 <1 
1 2/3. 1/3. 1/3 
1 1/3 1/3 2/3 
4 3 


all reduced costs of which are nonnegative. Finally, deleting the end column 
gives a dual feasible simplex tableau. 
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In this section, we present an artificial-variable free dual Phase-I method, involving 
neither minimum-ratio test, nor reduced costs. It uses the so-called most-obtuse- 
angle row rule, which is the dual counterpart of the most-obtuse-angle column rule 
(Sect. 14.4). 

Assume that the current dual solution is infeasible, and a pivot column index 
q € N is selected (e.g., by the conventional column rule) such that 


ig Sty cE R hay <0. (15.16) 


Lemma 15.4.1 Jf (15.16) holds and Ba, < 0, then the original problem is 
infeasible or unbounded. 


Proof Assume that x is a feasible solution. If we define/Defining 
dp=—-B ‘ag; dg=1; dj =0, jEN, 7 #a (15.17) 
the vector 
X=X+ad 
is a feasible solution for any a > 0; in fact, it holds that 
AX = Ax + a(Bdg + Ndn) = b, 


and, since x > 0 andd > 0, it holds that * > 0. It is known from (15.16) that the 
vector d satisfies 


cld = (chdp + chdw) = cq — chai = %q <0, (15.18) 


and hence the corresponding feasible value tends to 00 as @ tends to 00, i.e., 


T 


clk=cl 


xt+acid > -o, aw, 
indicating the lower unboundedness of the original problem. Therefore, the original 
problem is either infeasible, or feasible but unbounded below. Oo 


The key of this dual Phase-I method is to use the following row rule (Pan 1994, 
1997). 


Rule 15.4.1 (Most-Obtuse-Angle Row Rule) Select row index 


p € argmax{djg |i = 1,..., m}. (15.19) 
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Assume that dpg > 0. Taking a), as the pivot, carry out according elementary 
transformations. As a result, the negative reduced cost, indexed by g, becomes zero, 
whereas the zero reduced cost, indexed by j,, becomes positive, i.e., —Zg/Ap,q > 0. 

Overall steps can be summarized into the following algorithm. 


Algorithm 15.4.1 (Dual Phase-I: The Most-Obtuse-Angle Row Rule) Ini- 
tial: (B, N), B~!. This algorithm finds a dual feasible basic solution to the standard 
LP problem. 


1. Compute zy = cy — N'S, where y = BoTcp. 

2. Select pivot column index q such that g € arg minjey Z;. 

3. If Z_ = 0, compute xg = B'b, xn = 0, and stop. 

4. Compute dg = Ba. 

5. Stop if dg <0. 

6. Determine pivot row index p € argmax{dj, |i = 1,...,m.}. 
7. UpdateB™!. 

8. Update (B, N) by exchanging j, and q. 

9. Go to step 1. 


Theorem 15.4.2 If Algorithm 15.4.1 terminates at 


(i) step 3, then a dual feasible basic solution is obtained; or 
(ii) step 5, detecting infeasibility or lower unboundedness of the problem. 


Proof Termination at step 3 clear implies the current solution is dual feasible 
basic. When terminating from step 5, then it holds that zy < 0 anda, < 0; by 
Lemma 15.4.1, infeasibility or lower unboundedness of the problem is asserted. O 


Let us investigate the most-obtuse-angle row rule geometrically. For search 
vector d defined by (15.17), it holds that —a), < 0, and d forms the most-obtuse- 
angle with the gradient, e,, of the constraint Xj, = 0. In addition, (15.18) indicates 
that d is a descent direction with respect to the objective function c' x. If d is close 
to the negative objective gradient —c, therefore, then e, tends to form the most- 
obtuse-angle with —c. By the heuristic characteristic of the optimal basis (Part J), 
it is reasonable to let the constraint xj, > 0 be satisfied as equality, accordingly 
dropping xj, from the basis. 

This dual Phase-I method is advantageous for its remarkable simplicity and 
stability. Initially, it performed very well for solving a set of small test problems 
(Pan 1994). Taking MINOS 5.3 (Murtagh and Saunders 1998) as a platform, 
subsequently, a code based on it outperformed MINOS 5.3 (with default options) 
on 48 Netlib test problems with a CPU time ratio 1.37 (Pan 1997). 

Koberstein and Suhl (2007) extended this method to solve more general LP 
problems. They reported extensive computational results, which confirmed signifi- 
cant advantages of this method compared with some commonly used dual Phase-1 
methods. It was found, however, that the method performed a little unsatisfactorily 
on a few of the most difficult problems. 

Considering the extent to which the current vertex violates nonnegative con- 
straints, we propose the following variant to improve. 
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Rule 15.4.2 (Variant of Most-Obtuse-Angle Row Rule) Given constant0 < Tt < 
1. Select the pivot row index 


p €argmin{x;, | dig =t0,i=1,...,m}, O=max{ajg|i=1,...,m}> 0. 


In case of t = 1, the preceding is just Rule 15.4.1 itself. Practically, it should 
be close to 1 to expand the range of choices, so that when there exist negative 
components of xg, the nonnegative constraint that is violated the most is satisfied as 
equality. It may require around o = 0.95. 

Though Algorithm 15.4.2 uses Dantzig’s rule for column pivoting, rules pre- 
sented in Chap. 12 all apply. It has not been known which is the best. Intuitively, the 
most-obtuse-angle row rule seems to be best matched by the steepest-edge rule. But 
no numerical results are available at this stage. 

Note that the objective value does not necessarily monotonically change in 
the solution process. In addition, the finiteness of the algorithm is not guaranteed 
even under the degeneracy assumption. Guerrero-Garcia and Santos-Palomo (2005) 
offered a cycling instance. Even so, cycling has not been reported so far in practice. 


Example 15.4.1 Find a dual feasible simplex basis by Algorithm 15.4.1: 


min f = —5x, — 7x2 + x4, 

s.t. xy + 2x2 + x3 = 3, 
2x, + x2 —xX44+x5 = —2, 
—xy + x2 +4x44+%6= —-1, 
xj>0, jHl,...,6 


Answer Initial:B = {3,5, 6}, N = {1,2,4}, Bo! =7. 


Iteration 1: 


1. y= B-l'cg = (0,0,0)', zy = ey — N'y = (-5, -7, I)! 
2. min{z;|j = 1,2,4} = min{—5, —7, 1} = —7, q = 2, hence x2 enters the basis. 
4. a = B-!a) = (2,1, 1)". 
6. max{aj;2|i = 1, 2,3} = max{2, 1, 1} = 1, p = 1, so x3 leaves the basis. 
1/2 
7. Update B~'! = | -1/21 
-1/2 1 


8. B = {2,5,6}, N = {1, 3, 4}. 

Iteration 2: 

1. ¥ = (-7/2, 0,0)", zy = (—5, 0, 1)? — (—7/2, —7/2, 0)? = (—3/2, 7/2, 1)". 
. min{—3/2, 7/2, 1} = —3/2, q = 1, so x; enters the basis. 

. a = (1/2, 3/2, —3/2)7. 

. max{1/2, 3/2, —3/2} = 3/2, p = 2, so xs leaves the basis. 


or a) 
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1/3 1/2 23 =1/3 
7. Update B-!' = | 0 2/3 a1/21 LS | =17F 23 
Oo 1 1 hat? 4 ah 1 4 


8. B = {2, 1,6}, N = {5, 3, 4}. 
Iteration 3: 


1. } = (—3, -1,0)!, Zy = (0,0, 1)T — (-1, —3, 1)? = 1, 3, O)T. 

2.3 =q € argmin{1, 3, O}. 

3. Zj; = 0, Dual feasibility is achieved. According primal basic solution is 
xp = B-'b = (4/3, 1/3, —2)', xy = (0,0, 0)". 


The tableau version of Algorithm 15.4.1 is of a simple form. 


Algorithm 15.4.2 (Tableau Dual Phase-I: The Most-Obtuse-Angle Row Rule) 
Initial: simplex tableau of form (3.6). This algorithm finds a dual feasible simplex 
tableau. 


. Select pivot column index q € arg Minjen Z;. 

. Stop if z, = 0 (A dual feasible tableau obtained). 

. Stop if a, < 0 (infeasibility or lower unbounded). 

. Determine pivot row index p € argmax{ajg |i =1,...,m}. 

. Carry out elementary transformations to convert dp, to 1, and eliminate all other 
nonzeros of the column. 

6. Go to step 1. 


nABWN 
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Chapter 16 ®) 
Reduced Simplex Method od 


Consider the standard LP problem (12.1) with the additional assumption that the 
cost c is not a linear combination of rows of A. This is appropriate, since, otherwise, 
there exists a vector y such that c = A'y, leading to 


clx= yl Ax = y'b, 


which implies that the objective function is constant over the feasible region if any. 
Redefining the notation 


(A (0) 


we convert the standard problem into the so-called reduced problem below: 


min f 
s.t. (4| - enst) (7) =f 456, (16.1) 


where (A | —em41) € R&D *@+HD, b ER”, m <n, and rank A = m+1.The 
objective variable f is viewed as the (n + 1)th variable, corresponding to column 
Gn+1, Le., 


A A 
Te = Xn+1; An+1 = —em+1- 


The reduced problem is a special standard problem with a single free variable f 
as its objective. As a result, computations related to the simplex algorithm can be 
simplified as follows. 

Let (B, N) be a partition, where B is basis with the (m + 1)th basic column 


—em+1- 
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The reduced cost is then 
ZN= N'y, B'S = @m+1- 


If B = LU is available, the second system of the preceding can be converted to the 
two triangular systems below: 


Ulv= em+l> L'y =v. 


The solution to the first system is readily available, i.e., v = (1/)em+1, where pu is 
the (m + 1)th diagonal of U. Therefore, only a single triangular system 


LT¥ = (1/pwem+ 


needs to be solved to obtain the simplex multiplier y. In other words, only the three 
triangular systems are solved in each iteration, compared with the four systems in 
the simplex algorithm. However, the preceding tactic is not new since some LP 
codes, such as MINOS, already involve it. 

In this chapter, we introduce the so-called reduced simplex method. It is the first 
simplex method that searches along the objective-edge direction. Specifically, we 
put forward the reduced simplex algorithm and introduce its dual version with a 
dual Phase-I procedure in the next section. In the subsequent sections, we introduce 
several fusion versions of the primal and dual reduced algorithms, including the 
crisscross, perturbation, and bisection reduced simplex algorithms. In the final 
section, some notes are made on the birth of the reduced simplex algorithm. 


16.1 Reduced Simplex Algorithm 


The main feature of the introduced method is to use objective variable f as a special 
nonbasic variable, compared with the simplex method that always takes it as a basic 
variable. 

Let (B, N) be a pair of basis and nonbasis: 


B={ji,..-,jmei}, N=A\B, n+1¢B. (16.2) 


Assume that the equality constraint of (16.1) is converted to the following canonical 
form 


xp =b—Nxy — fan4i, (16.3) 
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where 
b= B'b, N=B!'N — Gntt = —B lems, #0. (16.4) 


Proposition 16.1.1 Assume that (x, f) with xy = 0 is a solution to the system 
(16.3). If there is a row index r such that 


xj, = 0, Gr, n+1 #0, (16.5) 
then x is a basic solution to the original problem. If, in addition, 
xB >= 0, (16.6) 


it is a feasible basic solution. 


Proof If (16.5) holds, x is just the basic solution corresponding to the simplex 
tableau, resulting from entering f to and dropping xj, from B. If, in addition, (16.6) 
holds, it is clearly a feasible basic solution. oO 


Consider the objective edge (Ax, Af) defined by 
AXxB = Gn+1, Af =-1, Axy = 0. (16.7) 


It is clear that the objective value reduces as it moves from x down along the 
edge. In general, however, it is not allowed to reduce infinitely to maintain the tg 
nonnegative. 

The largest possible a and associated blocking row can be determined by the 
following rule. 


Rule 16.1.1 (Pivot Row Rule) Assume that Gy41 % 0. Select the pivot row index p 
such that 
a= —Xj,/Gp,n41 = min{—X j;/i,n+1 | Gi,n+1 < 0, i= 1, a // i 1} = 0. 
(16.8) 
It is clear that the preceding rule is well-defined under its assumption. 
The corresponding new solution attained is as follows: 


fp=iptaings, f=f—a, tv=0. (16.9) 


We can claim the following. 


Lemma 16.1.1 Assume that x is the current feasible basic solution, and stepsize 
a is determined by (16.8). Then new solution X given by (16.9) is a feasible basic 
solution. 
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Proof By (16.8), it holds that 
$j,=0, pari <0, (16.10) 


and hence x is a basic solution. It is only needed to show that xg is nonnegative. 
Note that xg > O anda > 0. 
If di,n41 => 0 for some row index ij, it is clear that 


A 


Xj, = Xj, +OGj ny = 0. 
Now assume that row index 7 satisfies dj,n41 < 0. By (16.8), it is known that 
—@ > Xj,/Gi,n41- 
Multiplying the preceding by dj, p41 < 0 gives 
—adj, n+1— Xijis 
or equivalently, 
Xj; + 4j,n41 = O, 
which together with the first expression of (16.9) leads to 


x iz 0. 
Consequently, it holds that xg > 0, and x is therefore a feasible basic solution. 
i] 


Note that the stepsize a determined by (16.8) is equal to the decrement of the 
objective value. However, if the x ie 1s zero, so is the decrement. 

If some component of xg is zero, the feasible basic solution is said to be 
degenerate. 

It is obvious that the stepsize a vanishes only if degeneracy occurs. On the other 
hand, the w can be allowed to be infinite under a certain condition, as the following 
says. 


Lemma 16.1.2 [fdn41 > 0, problem (16.1) is infeasible or lower unbounded. 


Proof Assume that the feasible region of the problem is nonempty. Let (x, f) bea 
feasible basic solution, satisfying 


Xp =b— fani1 > 0, Xv =0. (16.11) 


We show that for any a > 0, the corresponding new solution given by (16.9) is a 
feasible solution. 
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Substituting 
pd = f — a, 


into the first expression of (16.9) gives 


gp =b— fans (16.12) 
= (b— fans1) + 0Gn41 (16.13) 
= Xp + Aan41 = 0, (16.14) 


where the inequality comes from (16.11) and a > 0 and the assumption a,+41 > 0. 
Therefore, the (x, f) is a feasible solution such that 


f=f—a— -—oo as a> Ow. 


This means that the problem is lower unbounded, and the proof is complete. oO 
Once row index p is determined, the following column rule applies. 


Rule 16.1.2 (Column Rule) — Select the pivot column index by 


iN Ap ;. 16.15 
qe aed ap, j ( ) 


Theorem 16.1.3. Under the same assumption of Lemma 16.1.1, if q is determined 
by (16.15) and Gp, q = 9, then X is an optimal basic solution. 


Proof According to Lemma 16.1.1, X is a feasible basic solution. Thus, it is only 
needed to show that f < f over the feasible region. 7 
From a@p,q = 0 and (16.15), it is clear that the pth row of N is nonnegative, i.e., 
elN > 0. (16.16) 
The equation corresponding to the pth row can be written 
Xj, = by — ey Nxn — fGp,n41. (16.17) 
From the first expressions of (16.12) and of (16.10), it follows that 
by = Papa ts 


Substituting the preceding into (16.17) gives 


nc Ts _ 
Xjp = fap, nti — e,Nxn — fap, n+1- 
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Multiplying the preceding by 1/dp,n41 and moving terms lead to the following 
expression of the objective function: 


f =f —(/ap.nse,Nan — (1/Ap,n+1)Xijp- (16.18) 


From (16.16) and the second expression of (16.10), it follows that the entries of 
the right-hand side of (16.18) are all nonnegative. Since the x is nonnegative over 
the feasible region, therefore, it can be asserted that < is an optimal solution with 
objective value f F Oo 


Starting from a feasible solution,! in each iteration, the algorithm determines the 
row index p and stepsize a, updates the solution, and determines column index q. 
If dp, ¢ = 0, optimality is achieved; otherwise, it makes the basis change and caries 
out the next iteration. 

These steps can be actualized with the so-called reduced simplex tableau, 
corresponding to the canonical system 


XB + Nxy + fans = b, 


that is, 


xe ty f | fat (16.19) 

I N an+1 XB 

which is related to the objective value f, corresponding to *g = b — fam. 
The overall steps can be summarized into the following algorithm. 


Algorithm 16.1.1 (Reduced Simplex Algorithm: Tableau Form) Initial: 


reduced tableau in form (16.19), with feasible solution (x, f). This algorithm 
solves (16.1). 


1. Stop if Gn41 = 0. 
2. Determine row index p and a such that 


a= —Xj,/Gp,n41 = min{—X j;/i,n+1 | Gi,n+1 < 0, i= 1,  // eo 1}. 


If a £0, update Xg = Xp + aan4 and f = f —a. 

Determine column index q € argMiNjen Gp j. 

Stop ifapq = 9. 

Convert Gpq to 1, and eliminate the other nonzeros in the pivot column by 
elementary transformations. 

7. Go to step 1. 


AwKRW 


' Tt is clearly not necessary to get started from a basic solution. 
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Note The elementary transformation in step 6 does not affect the xg column. 


Theorem 16.1.4 Under the assumption of nondegeneracy, Algorithm 16.1.1 termi- 
nates either at 


(i) step 1, detecting lower unboundedness of the problem, or at 
(ii) step 5, giving an optimal basic solution x. 


Proof According to Lemma 16.1.1, each iteration yields a feasible basic solution. 
Under nondegeneracy, stepsize @ is positive in each iteration. Therefore, the 
objective value decreases strictly monotonically, and the corresponding feasible 
basic solution never appears repeatedly. Since there are a finite number of such 
solutions, the algorithm must terminate. 
The meanings of the exits are obvious by Lemma 16.1.2 and Theorem 16.1.3. 
oO 


The preceding algorithm needs a Phase-I procedure to get it started. For this 
purpose, the generalized reduced simplex Phase-I Algorithm 18.1.3 should be 
a good choice. However, for standard LP problems, it seems preferable to use 
Algorithm 14.4.1 instead. 


Example 16.1.1 Solve the following problem: 


min f = 6x; + 4x2 — 6x3 — 4x4, 


st. 3xy +2x2+3x3+5x4+ x5 =: 3; 
—4x, —x2+ 5x3 -— 2x4 + x6 = 2; 
—6x; —7x2 —1x3+4x4 + 4x7 = —5, 

x, — 3x2 -—2x3 —Xx4 +xg=-l, 


xj 20, jHl,...,8. 


Answer Phase-I: Call Algorithm 14.4.1. 
Initial tableau is 


Bal x2 X3 X4 X5 X6 X7  Xg f | RHS 
3 1 3 5 1 3 

-4 -1 5 —2 1 2 

-6 -—7 -l 4 1 —5 
1 —-3 —2 -!l 1 -1 
6 4 —6* —4 -1 


Iteration 1: 


1. min {3, 2, —5, -—1} = —5, p =3. 
3. min {—6, —7, —1, 4} = —7, q =2. 
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5. Multiply row 3 by —1/7, and add —1, 1, 3, —4 times of row 3 to rows 1, 2, 4, 5, 
respectively: 


xX] x2 X3 X4 X5 X6 x7 x8 
15/7 20/7 39/7 1 1/7 
22/7 36/7 —18/7 i 1/7 
6/7 1 1/7 —4/7 -1/7 
25/7 =1ij7 1977 3/7: 4 
18/7 =46/7 =12/7 4/7 


The preceding tableau is feasible. Convert it to the reduced simplex tableau: 
Iteration 2: 


1. p=5, min{18/7, —46/7, —12/7, 4/7} = —46/7 <0, q =3. 


2. Multiply row 5 by —46/7, and add —20/7, —36/7, —1/7, 11/7 times of row 5 
to rows 1, 2, 3, 4, respectively: 


Xx] X20 X3 x4 Xs XG x7 xg f f=0 
75/23 111/23. 1 9/23 —10/23 | 24/23 
~26/23 —90/23 1 7/23 —18/23 | 11/23 
21/23 1 ~14/23 ~3/23 -1/46 | 15/23 
68/23 —53/23 ~13/23 1 11/46 | 42/23 
—9/23 1 6/23 —2/23 7/46 10/23 


Phase-II. Call Algorithm 16.1.1. 
Iteration 3: 


1. ay ZO. 
2. a = min{—(24/23)/(—10/23), —(11/23)/(—18/23), —(15/23)/(—1/46)} 
= (11/23)/(—18/23) = 11/18, p =2. 
. ¥g = (7/9, 0, 23/36, 71/36, 19/36)", f = 0— 11/18. 
. min{—26/23, —90/23, 7/23} = —90/23 <0, q =4. 
6. Multiply row 2 by —23/90, and add —111/23; 14/23; 53/23; —6/23 times of 
row 2 to rows 1, 3, 4, 5, respectively: 


RW 


XxX] x2 X3 X4 X5 X6 x7 
28/15 1 37/30 ~——-23/30 
13/45 1 —23/90 7/90 
49/45 1 -7/45 8/45 

163/45 —53/90 —67/90 


—T/15 1 1/15 —1/15 
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Iteration 4: 


1. Gp # 0. 
2. w = min{—(7/9)/(—7/5)} = 5/9, p = 1. 
3. Xp = (0, 1/9, 25/36, 85/36, 7/12)", f = —11/18 — 5/9 = —7/6: 


Bal xX. 2X3 X45 X6 x7 xg f | f = -—7/6 
28/15 1 37/30 23/30 7/5 | 

13/45 1 —23/90  —7/90 1/5 1/9 
49/45 1 -7/45 —8/45 1/10 25/36 
163/45 —53/90 -67/90 1 7/10 85/36 
-7/15 1 1/15 1/15 1/10 | = 7/12 


4. min{28/15, 37/30, 23/30} > 0. 
5. The optimal basic solution and optimal objective value: 


x* = (0, 25/36, 7/12, 1/9, 0, 0, 0, 85/36)!, f* =-7/6. 


Let B be the current basis. Corresponding to the reduced simplex tableau (16.19), 
the revised tableau is 
xe ty _f _|f=f (16.20) 
I BUN —Bem+1| XB 


Based on the equivalence between the two tableaus, it is not difficult to transform 
Algorithm 16.1.1 into its revised version. 


Algorithm 16.1.2 (Reduced Simplex Algorithm) Initial : (B, N), Bo! xg = 
b — fan41 => 0, where b = Bob, an41 = =e Bees aie, This algorithm solves 
reduced problem (16.1). 


1. Stop if Gn41 => 0 (unbounded problem). 
2. Determine a and row index p such that 


a= —Xj,/Qp. nti = min{—X j;/i, n+1 | Gi,n+1 < 0, c= 1, a // i 1}. 
. Ifa £0, update Xp =Xgp+aani1, f = f—a. 
. Compute oy = NIB Te, 
Determine column index q € argminjey Oj. 
. Stop if og = 0 (optimality achieved). 
Compute ag = Bag. 
Update Gn41 = Gn41 + V(Ag — @p), where v = —Ap,n+1/0q. 
. Update B~!. 


OC mMNAWAYW 
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10. Update (B, N) by exchanging jp and q. 
11. Go to step 1. 


16.2 Dual Reduced Simplex Algorithm 


The dual version of the reduced simplex algorithm can be obtained similarly to 
the derivation of the classical dual simplex algorithm; that is, using the same 
reduced simplex tableau (16.19), we seek primal feasibility while maintaining dual 
feasibility, until attaining optimality. 

The reduced simplex tableau is said to be dual feasible if the following condition 
is satisfied for some row index p: 


eIN>0, Gp,nti <0. (16.21) 


The associated basic solution (x, f) is such that 
f=bp/apnu, te=b—fany1, in =, (16.22) 
with 
Xj, = 9. (16.23) 


The pth row of the tableau is termed the objective row. It corresponds to the 
equation 


Xjp = bp = e,Nxn — fap,ntis 
or equivalently, 
= bal Op aaa = (1/ap, nile, Nx —Xj,/Ap,n+1- (16.24) 


Consider the following rule. 
Rule 16.2.1 (Dual Row Rule) Select the row index r such that 
xj, =min{x; |i=1,...,m+1; iF ph. 
According to Theorem 16.1.3, (x, f) is an optimal basic solution if x;, = 0, and 
hence xz > 0. 


Now assume the opposite, i.e., 


xj, <0. (16.25) 
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Then we have the following. 


Lemma 16.2.1 Assume the satisfaction of (16.21), (16.22), and (16.25). The LP 
problem is infeasible if it holds that 


J={ZEN |G; <O}=8, nti = 0. (16.26) 
Proof There are only the following two cases: 


(i) ar, n+l = 0. 
In this case, it holds that 


x; =b, —e) Nz, (16.27) 
substituting (x, f) into which gives 


x; = b, <0, (16.28) 


where the inequality is from (16.25). Suppose that there is a feasible solution 
(x’, f’). Then, it satisfies (16.27), ie., 


/ L. Tazy,/ L, 
x, = by — e, Nxy X< by, 


where the inequality is from x), > 0 and the first expression of (16.26). From the 
preceding and (16.28), it follows that x’, < 0, contradicting that (x’, f’) is feasible 
solution. Therefore, the is exists no feasible solution. 

(ii) Gp,n41 > 0. 

By the second expression of (16.22), it holds that 


hj = bp — far, wit, 

which along with (16.25) and a, 41 > 0 gives 

F = br/Gr, ni — ¥j,/Gr,n41 > br /Gp, n41- (16.29) 
In addition, the equation corresponding to the rth row is 

xj, = b, _ e; Nxn — fay, n+1, 
or equivalently, 
f =br/apnsi — (1/a, neve; Nxn — Xj, /Gp,n41- 

Suppose that there is a feasible solution (x’, f’). Then it satisfies the preceding, i.e., 


Ff) = by fap nt — far, npidep Nay — x), [ar ntl. 
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From the preceding, a,,,4+1 > 0 and the first expression of (16.26), it follows that 
Fl Sp far, nt, 

which along with (16.29) leads to 

fe biG ft (16.30) 
On the other hand, substituting (x’, f’) into (16.24) leads to 
f! = bp/Gp,n41 — (1/Gp,neegNxy — x4, /Gp,n41- 

The preceding with (16.21) and the first expression of (16.22) gives 
2s, 

which contradicts (16.10). Therefore, there exists no feasible solution. oO 


The following rule is well-defined. 


Rule 16.2.2. Assume that the set J defined by (16.26) is nonempty. Select column 
index q such that 


B = Gp. q/tir.g = min —Gp, j/t, j = 0. (16.31) 


To make the corresponding basis change, multiply the rth row by 1/a,, g, and add 
—dj,q times of the rth row to the ith row, fori = 1,...,m-+1,i #r. Then, update 
the column index sets: 


B= B\{j-} U {q}, N= N\{q} U {jr}. (16.32) 


Note that the pth row of the resulting tableau yields from adding f times of the 
r row to the old pth row. Therefore, it can be denoted by 


Xjp XN_ _ f poluian (16.33) 
1 elN Gp,nti| Xjp 
where 
Gp,nt1 = 4p,n41 + Bar, n+1, Xj, = BX;,, (16.34) 


with the nonbasic coefficients omitted. 
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If 6 # 0 and hence ip © 9, the resulting is clearly not a dual reduced simplex 
tableau yet. In this case, we add 


a = —£),/Ap, n41 (16.35) 


times of the f column to the tg column, which corresponds to updating the 
objective value by 


pape 


If the dp, q is zero, so is the 6 defined by (16.31); therefore, from (16.34) and 
(16.35), it follows that the a vanishes. Consequently, the new objective value f is 
not really new, but the same as the old. 

If e'N > 0, the dual reduced simplex tableau is said to be dual nondegenerate. 

Under the assumption of nondegeneracy, the objective value increases strictly. 

Then, we can claim that the end tableau is as follows. 


Lemma 16.2.2. Under the same assumption of Rule 16.2.2, there is no feasible 
solution if Ap, n+1 = 0. 


Proof In this case, it is verified that elN > 0. Therefore, the claim can be shown 
as the same as that for Lemma 16.26. oO 


Based on the preceding discussions, the validity of the following result is 
obvious. 


Lemma 16.2.3 Under the same assumption of Rule 16.2.2, if Gp,n+1 < 0, the new 
reduced simplex tableau is a dual feasible tableau with the pth objective row and 
the objective value f. 


Overall steps can be summarized into the following. 


Algorithm 16.2.1 (Dual Reduced Simplex Algorithm: Tableau Form) Initial 
reduced simplex tableau (16.19), where enN = 0, 4p nti < 9, f = 


b p/p, ntl, XB = b- fan4t. This algorithm aoe the reduced problem (16.1). 

1, Determine row index r € argmin{x;, |i=1,...,m+1,i 4 p}. 

2. Stop if x;, = 0 (optimality achieved). 

3. If J ={j © N | a,j < 0} =Y, set p =r, and go to step 6. 

4. Determine B and column index q such that B = —Gpq/@rqg = MiNjey —Ap;/A,j.- 
5. Convert a; q to 1, and eliminate the other nonzeros of the column by elementary 


transformations. 

Stop if Gp, n41 = 0 (infeasible problem). 

. Compute a = —xj,/Ap,n4+1- 

. Ifa £0, update Xp = Xp + aGn4, and f = f —a. 
. Go to step 1. 


6 MO ND 
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Theorem 16.2.4 Under dual nondegeneracy, Algorithm 16.2.1 terminates. It ter- 
minates at 


(i) step 2, giving an optimal basic solution x; or at 
(ii) step 6, detecting the infeasibility of the problem. 


Proof Since there exist only a finite number of bases, the algorithm does not 
terminate implies that some bases are repeated infinitely many times, which 
contradicts that the objective value increases strictly monotonically in the solution 
process, under the dual nondegeneracy. Therefore, the algorithm terminates. 
According to the Lemmas 16.2.1, 16.2.2, 16.2.3, and Theorem 16.1.3, the 
meanings of the exits of the algorithm are obvious. Oo 


Example 16.2.1 Solve the following problem by Algorithm 16.2.1 : 


min f = 4x, + 2x2 + 3x3 + 8x4 + 5x5, 


st. —x, + 5x2 — 4x4 — 2x5 + x6 = —3, 
—3x, —2x2 -— 6x3 +2X4— x5 + x7 =-l, 

— x2 + 4x3 — 6x4 + 4x5 + xg = 5, 

5x1 + 3x2 — 3x3 + 3x4 + 5x5 +x9= 2, 


xj>0, j=1,...,9. 


Answer Initial tableau: 


Xp x2 x3 Xq XS XH XT XQ og | Of | f =0 
-1 5 -4 -2 1 3 
—3 —2 -6 1 -l 1 -l1 
-1 4 —-6 1 5 
5 3 -3 3 5 1 2, 
2* 3 8 5 —1 


Iteration 1: 

To convert the preceding tableau to the reduced simplex tableau, take the smallest 
2 among the first 5 entries in the bottom row (in x2 column) as the pivot. 

Multiply row 5 by 1/2, and add —5, 2, 1, —3 times of row 5 to rows 1, 2, 3, 4, 
respectively. 

Take p = 5,esN > 0, Gs,19 = —1/2 < 0. xg = (3, —1,5, 2, 0)!: 


Xp X2 x3 x4 x5 6 4X7 «xg x9 | Of f=0 
-l1 —15/2 —24* —29/2 1 | 5/2 3 
1 3 9 4 1 | -1 -l1 
2 11/2 2 13/2 1 l-in| 5 
== 15/2 -9 —5/2 1 | 3/2 2 
2 1 3/2 45/2 | 472 
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Call Algorithm 16.2.1. 


Iteration 2: 


1 


5. 


6. 
. a = —(—1/2)/(—1/12) = -6. - 
. XB = (3/4, -7/4, 19/2, -1/4, 0)", f = 0- (-6) = 6. 


~ 


XxX] x2 X3 X4 X5 X6 x7 X8 
11/24 5/16 1 29/48 —1/24 
~25/8 —93/16* 23/16 3/8 1 
35/12 49/8 185/24 —1/12 1 
25/8 ~75/16 47/16 —3/8 
1/6 1 1/4 1/12 1/6 


. min{—3, —1, 5, 2,0} = -3 <0, r=1. 
3. 
4. 


J={1,3,4,5} 49. 
B = min{—2/(—11), —G3/2)/(—15/2), —4/ — 24, —(3/2)/(—29/2)} 
= —4/-—24= 1/6, q =4. 
Multiply row | by —1/24, and add —9, 2, 9, —4 times of row | to rows 2, 3, 4, 5, 
respectively: 
45,19 = —1/12 <0. 


Iteration 3: 


WwW 


xX] x2 x3 X4 x5 X6 XT x8 
9/31 1 49/93 —2/93 5/93 
50/93 1 23/93 2/31 —16/93 
—35/93 192/31 29/93 98/93 1 
175/31 127/31 —21/31 =25/61 


_ min{3/4, —7/4, 19/2, -1/4} = -7/4 < 0, r =2. 
J = {13,5} £0. 
. B = min{—(1/6)/(—25/8), —(1/4)/(—93/16), —(./12)/(—23/16)} 


= —(1/4)/(—93/16) = 4/93, q =3. 


. Multiply row 2 by —16/93, and add —5/16; —49/8; 75/16; —1/4 times of row 


2 to rows 1, 3, 4, 5, respectively: 


: a5,10 = —7/93 <0. 
. a = —(—7/93)/(—8/93) = —7/8. 7 
. XB = (3/4, 7/24, 25/3, 5/8, 0)? column, and set f =6-— (-7/8) = 55/8: 


1/31 1 2/93 17/93 4/93 
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Iteration 4: 


1. min{3/4, 7/24, 25/3, 5/8} > 0. 
2. The optimal basic solution and optimal value: 


x* = (0, 0, 7/24, 3/4, 0, 0, 0, 25/3, 5/8)', f* =3x(7/24)8x (3/4) = 55/8. 
Based on the equivalence between the reduced tableau (16.19) and the revised 


tableau (16.20), the following revision of Algorithm 16.2.1 can be obtained. 


Algorithm 16.2.2 (Dual Reduced Simplex Algorithm) — Initial: (B, N), Bo, 
o =e BN > 0 Gone < 0, f = Bp/apnvi, tp = 6 — fans = 0, 


where b = Bop, ani = Bo Gai. This algorithm solves the reduced problem 
(16.1). 

1. Determine row index r € argmin{x;, |i =1,...,m+1, iF p}. 

2. Stop if x;, > 0 (optimality achieved). 

3. Compute oy = N'BoTe,. 

4. IfJ ={j €N | aj <0} =G, set p=r, on = on, and goto step 11. 

5. Determine B and column index q such that B = —0g/@q = minjey —0j/@j;. 

6. Compute ag = Bo lag. 

7. Update :Gn41 = Gn41 + v(dg — er), where v = —Gy,n41/@gq- 

8. Update :Xg = Xp +t(dq — er), where t = —Xj,/gq. 

9. Update B~'. 


10. Update (B, N) by exchanging jp and q. 

11. Stop if @p,n41 = 0 (infeasible problem). 

12. Compute a = —xj,/Ap,n+1- 

13. Ifa £0, update Xp =Xpt+aani1, f= f—a. 
14. Go to step 1. 


16.2.1 Dual Reduced Simplex Phase-I: Most-Obtuse-Angle 


Based on the most-obtuse-angle heuristic, the overall steps can be put into the 
following algorithm. 


Algorithm 16.2.3 (Dual Reduced Phase-I: Tableau Form) — Initial : reduced 
simplex tableau in form (16.19), where xp = b — fan41. This algorithm finds a 
dual feasible reduced tableau. 


. Determine row index p € arg min{dj,n+1 |i = 1,...,m + 1}. 
. Stop if Ap,n41 = 0 (infeasible or unbounded problem). 
Compute a = —xj,/Gp,n+1- 

. Ifa £0, update xp = Xp +aay+1 and f = f =i. 

. Determine column index q € argMinjen Gp j- 

. Stop if Apq = 0 (dual feasibility achieved). 
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16.2 Dual Reduced Simplex Algorithm 419 


7. Convert ap q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
8. Go to step 1. 


Note Step 7 is independent of the xg column. 


Example 16.2.2 Solve the following problem using Algorithm 16.2.3 as dual 
Phase-I : 


min f = —4x1 — 7x2 + 2x3 + 3x4, 


s.t. —2x, + lxo —3x3- x4 + x5 =-ll, 
3x, + 3x2 + x3 4+ 5x4 + x6 = 35, 

5x1 + 4x2 — x3 -— 2x4 + x7 — 2. 

— xy + 6x2 — 7x3 + 4x4 t+x= 7, 


xj>0, jHl,...,8. 


Answer Initial tableau: 


where xg = (—11,35,2,7)', f =0. 
Dual Phase-I: call Algorithm 16.2.3. 
Iteration 1: 


. min{0, 0,0,0, -1} = -1 <0, p=5S. 

.a=0/-1=0. 

. min{—4, —7, 2,3} = —7 <0, q =2. 

. Multiply row 5 by —1/7, and add —1, —3, —4, —6 times of row 5 to rows 
1, 2, 3, 4, respectively: 


NW Re 


x] x2 X3 x4 x5 X6 X7 
—18/7 —19/7 —4/7 1 
9/7 13/7 44/7 1 
19/7 1/7 -2/7 1 
—31/7 —37/7* 46/7 


4, 1 la 
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Iteration 2: 


NU Rw 


min{—1/7, —3/7, —4/7, —6/7, 1/7} = —6/7 <0, p=4. 

a = —7/(—6/7) = 49/6. 

Xp = (—73/6, 63/2, —8/3, 0,7/6)', f = —49/6. 

min{—31/7, —37/7, 46/7, 0, 7/6} = —37/7 < 0, q =3. 

. Multiply row 4 by —7/37, and add 19/7, —13/7, —1/7, 2/7 times of row 4 to 
rows 1, 2,3, 5, respectively: 


Xx] x2 X3 X4 X5 x6 x7 

—11/37 — 146/37 1 

—10/37* 318/37 1 
96/37 —4/37 1 
31/37 1 —46/37 
30/37 1 —29/37 


Iteration 3: 


NOB We 


. min{11/37, —27/37, —22/37, 6/37, 7/37} = —27/37 < 0, p =2. 

. & = —(63/2)/(—27/37) = 259/6. 

. ¥g = (2/3, 0, , 85/3, 7, 28/3)", f = —49/6 — 259/6 = —154/3. 

. min{—10/37, 318/37, 13/37} = —10/37 < 0, gq =3. 

. Multiply row 2 by —37/10, and add 11/37, —96/37, —31/37, —30/37 times of 
row 2 to rows 1, 3, 4, 5, respectively: 


Xp x2 x3 X4 x5 X6 x7 
-67/5 1  —11/10 
1 —159/5 —37/10 
412/5 48/51 
1 127/5 31/10 
1 25 3 


Iteration 4: 


1S’) 


nN 


. min{11/10, 27/10, —38/5, —21/10, —2} = —38/5 <0, p =3. 
_ a@ = —(—85/3)/(—38/5) = —425/114. 
. ¥g = (—261/76, —765/76, 0, 1127/76, 319/19)", % = —154/3 + 425/114 = 


—1809/38. 


. min{412/5, 48/5, 17/5} => 0. 
. Dual feasibility is achieved: 


Phase-II: call Algorithm 16.2.1 
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XxX. x2 X3 x4 X5 X6 x7 f = —1809/38 
-67/5 1  —11/10 —261/76 
1 —159/5* ~37/10 ~765/76 
412/5 48/51 
1 127/5 31/10 1127/76 
1 25 3 319/19 


Iteration 5: 


1. min{—261/76, —765/76, 1127/76, 319/19} = —765/76 < 0, r = 2. 
3. J = {4,6, 8} 4G. 
4. B = min{—(412/5)/(—159/5), —(48/5) /(—37/10), —(17/5)/(—13/10)} 
= (412/5)/(159/5) = 412/159, gq = 4. 
5. Multiply row 2 by —5/159, and add 67/5, —412/5, —127/5, —25 times of row 
2 to rows 1,3,4,5, respectively: 


x1 x2 X3 X4 X5 X6 XT xg f = — 1809/38 
—67/159 1 = 73/159 —56/159 —2/53 813/1007 
—5/159 1 37/318 13/318 —9/106 358/1131 
412/159 2/159 1 5/159 | —32/53 | —4747/182 
127/159 1 23/159 —22/159 3/53 4311/635 
125/159 1 29/318 —7/318 13/106 6950/783 


6. 43,9 = —32/53 <0. 

7. @ = —(—4747/182)/(—32/53) = —8683/201. 

8 .Xp = (39/16, 255/64, 0, 139/32, 229/64)', f = —1809/38—(—8683/201) = 
—16898/3835: 


Xx] X. 3 x4 x5 X6 x7 xg f = —16898/3835 

—67/159 1 73/159 —56/159 | —2/53 39/16 
—5/159 1 37/318 13/318 | —9/106 255/64 
412/159 2/1591 5/159 | —32/53 

127/159 1 23/159 22/159 3/53 139/32 
125/159 1 29/318 -~7/318 13/106 229/64 


The optimal basic solution and optimal value: 


x* = (0, 229/64, 139/32, 255/64, 39/16, 0, 0, 0), 
f* = —16898/3835. 
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The following is a revision of Algorithm 16.2.3. 


Algorithm 16.2.4 (Dual Reduced Phase-I: Most-Obtuse-Angle) Initial 
(B,N), Bo!. xB = b— fGni1, where b = Bob, any, = —Bo!em41. This 
algorithm finds a dual feasible basis to (16.1). 


1. Determine row index p € arg min{dj,n41 |i = 1,...,m-+ 1}. 
2. Stop if Gp,n41 = 0 (infeasible or unbounded problem). 
3. Compute a = —Xj,/Ap,n+1- 

4. Ifa £0, update Xg =Xptaany1, f = f—a. 

5. Compute on = NIB Te, 

6. Determine column index q € arg minjen Gj. 

7. Stop if og = 0 (dual feasibility achieved). 

8. Compute ag = Boag. 

9. Update an41 = Gn41 + V(dg — @p), where v = —Ap,n41/0q- 
10. Update B=. 
11. Update (B, N) by exchanging jp and q. 
12. Go to step 1. 


16.3 Perturbation Reduced Simplex Algorithm 


As it is known, the simplex method is not finite. Degeneracy often leads to stalling 
and therefore takes too much time in the solution process although cycling hardly 
happens in practice. The perturbation simplex method (Pan, 1999, 2000) aims to 
eliminate this defect by taking strictly positive stepsizes. 

The degeneracy in the reduced simplex context causes similar problems. As 
a remedy, in this section, we realize the same idea to introduce the so-called 
perturbation reduced simplex algorithm. Starting from any basis, it performs 
the normal reduced (or dual reduced) simplex algorithm but with the negative 
components of each solution perturbed to some small positive numbers. Actually, it 
solves another problem with the right-hand side b (reduced cost c) modified and 
finally recovers and solves the original problem with the dual (primal) reduced 
simplex algorithm. If the changes are small enough, there will be no need for 
performing the latter reduced simplex algorithm. 

The overall steps are put into the following algorithm. 


Algorithm 16.3.1 (Perturbation Reduced Simplex Algorithm: Tableau Form) 
Given perturbation parameters 6; = 1, j = 1,...,n. Initial: reduced tableau of 
form (16.19), where xp = b — fan+1. This algorithm solves the reduced problem 
(16.1). 


1. Call the reduced simplex Algorithm 16.1.1 but use 4;, instead of xj, in the 
minimum-ratio test (16.8) if x;, < 6;,, i= 1,...,m,. 
2. Stop if returning from step I (infeasible problem). 
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3. Returning from step 5: stop if xg > 0 (optimality achieved). 
4. Call the dual reduced simplex Algorithm 16.2.1 but use 5; instead of dp, ; in the 
minimum-ratio test (16.35) if ap, j < 4j. 


5. Stop if returning from step 6 (infeasible problem). 
6. Returning from step 2: stop if ap, ; = 9, j € N (optimality achieved). 
7. Go to step I. 


Example 16.3.1 Solve the following problem by Algorithm 16.3.1: 


f = 2x1 + 7x2 — 5x3 — 3x4, 


+X6 


min 
s.t. —x, +4x2 —5x3 —2x4 +25 
xy —3x2 +3x3 5x4 
7x, —6x2 —x3 —4x4 
—3x, +x. +2x3 —Xx4 
xj>0, j=l,...,8. 


Answer Set perturbation parameters 5; = 1, j =1,... 


1. Call Algorithm 16.1.1. Start from the initial tableau: 


X3 X4 x5 
=2. 1 
5 
—4 
1 
—3 


. min{—2, 7, —5, —3} = —5 <0, g =3. 
. Multiply row 5 by —1/5, then add 5, —3, 1, —2 times of row 5 to rows 1, 2, 3, 4, 


x3 X4 X5 
1 1 
16/5 
sie 
-11/5 


xX) XQ 
-1 4 
1 3 
a —6 
3 i 
—2 7 
Iteration 1: 
1. ag = —e5 F O. 
2.a=0/(-1) =0, p=5. 
4 
6 
respectively. 
x1 x2 
1 —3 
-1/5 6/5 
37/5 —37/5 
-19/5* 19/5 
2/5 7/5 


1 3/5 


+X7 = 


+xXg = 


8. 
f =0. 


423 
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Iteration 2: 


2s 


3: 
4. 
. Multiply row 4 by —5/19, then add —1, 1/5, —37/5, —2/5 times of row 4 to 


Determine row index by replacing the right-hand side term —2 by 
I:min{—2/(—3/5), -1/(—2/5)} = 5/2, p = 4,0 = —(—2)/(—2/5) = —S. 


tp = (7,5, -16,0,—1)', f =0—(—5) =5. 
min{—19/5, 19/5, 11/5} = -19/5 <0, q=1. 


rows 1, 2, 3, 5, respectively: 


Xt X23 x4 Xs x6) XT xg f 
—2 8/19 1 5/19 17/19 
1 63/19 1 —1/19 —11/19 
—146/19* 1 37/19 —11/19 
i <4 11/19 5/19 2/19 
= ae 7/19 2/19 3/19 


Iteration 3: 


2. 


B&R wW 


Determine row index by replacing the right-hand side term — 16 by 1: 


a = min{—5/(—11/19), —1/(—11/19)} = 19/11, p=3; w = —(—16)/(—11/19) = —304/11. 


. ¥g = (—195/11, 21, 0, —32/11, —59/11)", f = 5 — (—304/11) = 359/11. 
. min{0, —146/19, 37/19} = —146/19 <0, g =4. 
. Multiply row 3 by —19/146, then add —8/19, —63/19, —11/19, —7/19 times 


of row 3 to rows 1, 2, 4, 5, respectively: 


x] xX. X3  X4 XS XG x7 xg = 359/11 
—2 1 4/73 27/73 —195/11 
1 1 63/146 115/146 | —121/146 21 
1 —19/146 —37/146 11/146 
-1 11/146 —17/146 9/146 —32/11 
-1 1 7/146 29/146 19/146 —59/11 


Iteration 4: 


2. 
3: 


4. 
3. 
4. 


a = min{—21/(—121/146)} = 3066/121, p = 2. 

xp = (501/121,0, 21/11, —163/121,—250/121)', f = 359/11 — 
(3066/121) = 883/121. 

min{1, 63/146, 115/146} > 0. 

Returning from step 5, but xg 7 0. 

Call Algorithm 16.2.1. 
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7 = 38/01 


Pal X2 X33 X4 = =xX5) (XG x7 xg 
= 1 4/73 27/73 63/73 501/121 
1 1 63/146 115/146 | —121/146 
1 19/146 —37/146 | 11/146 21/11 
i <7 11/146 -17/146 | 9/146 ~163/121 
-1* 1 7/146 29/146 | 19/146 ~250/121 


Iteration 5: 


. min{501/121, 0, 21/11, —163/121, —250/121} = —250/121 < 0, r=5S. 

J = {23} 490. 

. minf—1/(-—D} = 1,q =2. 

. Multiply row 5 by —1, then add 2,—1,1 times of row 5 to rows 1, 2, 4, 
respectively: 


an BW Re 


Xp x2 X3 = X4 = X5 XG x7 xg RHS 
—2 1 —3/73 —2/73 44/73 91/11 
1 1 35/73 72/73 —51/73 —250/121 
1 —19/146 —37/146 11/146 21/11 
1 —1 2/73 —23/73 —5/73 87/121 
1 -l —7/146 —29/146 | —19/146 250/121 


6. G2,9 = —51/73 <0. 

7. « = —(—250/121)/(—51/73) = —1943/657.. 

8. gp = (331/51, 0, 86/51, 47/51, 121/51)", f = 883/121 — (—1943/657) = 
523/51. 


Xt = X2 x3 4X4 «2X5 XG x7 xg f = 523/51 
—2 1 —3/73 —2/73 44/73 331/51 
1 1 35/73 72/73 —51/73 
1 —19/146 —37/146 11/146 86/51 
1 -1 2/73 —23/73 —5/73 47/51 


1 -l —7/146 —29/146 | —19/146 125/51 
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Iteration 6: 


1. min{331/51, 0, 86/51, 47/51, 125/51} > 0. 
6. Returning from step 2, and a2; > 0, j € N. The optimal basic solution and 
optimal value: 


x* = (47/51, 125/51, 0, 86/51, 331/51, 0, 0,0)", f* = 523/51. 


Although the preceding problem was solved by calling the primal and dual 
reduced simplex algorithms once each, there would be more calls required for 
solving large-scale and highly degenerate problems. 


16.4 Bisection Reduced Simplex Algorithm 


In some applications, one can estimate the optimal value to some extent. In this case, 
it would be possible to speed up the solution process. If the optimal value is known 
to be y, e.g., after adding the extra constraint c'x = y, any feasible solution to the 
modified problem is clearly an optimal solution to the original problem. Therefore, 
only a Phase-I procedure is needed to solve the problem. Although this is not the 
case in general, it would be possible to determine a so-called existence interval, 
including the optimal value within. It is clear that the optimal value divides the 
interval into two subintervals: the left-hand one consists of dual feasible values 
while the right-hand one consists of primal feasible values. 

This is the basic idea behind the bisection simplex algorithm Pan (1991, 1996). 
Let [A, 2] be an existence interval. The algorithm seeks a feasible solution attaining 
the objective value 


F=Qty)/2, 


using the subalgorithm similar to the infeasibility-sum Phase-I (Sect. 14.1). 
The outcome will be one of the following cases: 


(i) Such a feasible solution is found, hence the associated objective value f is 
a feasible value. We shift it down to the objective value corresponding to 
a feasible basic solution. If the dual feasibility is also attained, so is the 
optimality; otherwise, its objective value is a new upper bound on the optimal 
value and hence is set to ju. 

(ii) There is no such feasible solution, hence the reached solution is dual feasible. 
We shift it up to the objective value corresponding to a dual feasible basic 
solution. If the primal feasibility is reached, so is the optimality; otherwise, 
its objective value is a new lower bound on the optimal value and hence is set 
tor. 
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Consequently, a new existence interval [A, jz] is obtained, contracted by half of the 
previous length, at least. This process is repeated until the length of the interval 
becomes so short that an optimal solution can be found. 

Just recently, the author was surprised to find that the ‘second type of simplex 
tableau’ used in the bisection algorithm is the same as the reduced simplex tableau. 
In this section, we introduce the algorithm simplified by using the most-obtuse-angle 
column rule (Sect. 14.4). Moreover, we use the following approach to produce the 
initial existence interval [A, jx]: 

Firstly, the desirable Phase-I (Algorithm 14.4.1) is called to seek the primal 
feasibility. If the solution reached is feasible, its objective value is set to jz. Then, 
starting from the end basis, the dual reduced Phase-1 Algorithm 16.2.3 is called to 
seek the dual feasibility. If the solution reached is dual feasible, its objective value 
is set to A. Therefore, the interval (A, 4) can be determined unless the problem is 
found to be infeasible or lower unbounded; and the length of the interval might not 
be too large. 

The overall steps are summarized into the following algorithm (without proofs). 


Algorithm 16.4.1 (Bisection Reduced Simplex Algorithm: Tableau Form) /ni- 
tial: Reduced simplex tableau of form (16.19) with f and xg = b — fan41. This 
algorithm solves the reduced problem (16.1). 

Step 1. Call Algorithm 14.4.1 to achieve feasibility: 

(1) set u = f if returned from step 4; 

(2) stop if returned from step 2 (the feasibility is attained). 

Step 2. Call Algorithm 16.2.3 to seek the primal feasibility: 

(1) stop if returned from step 2 (lower unbounded problem); 

(2) setA = ff if returned from step 6. 

Step 3. Compute f =(A+u)/2,a=f-— f. 

Step 4. Update Xp = Xp + n+ and f = 7: 

Step 5. Determine row index p € argmin{x;, |i = 1,...,m+ 1}. 

Step 6. If xj, = 0 then: 

(1) determine a and row index p such that 


a= —Xj,/Ap, nti = min{—x j;/Gi,n+1 | Gi,n+1 < 0, i= 1, a // ie 1}; 


(2) update Xp = Xp +aGp41 and f = f —a; 

(3) determine column index q € argminjeNn 4p, j; 

(4) stop if p,q = 9 (optimality achieved); 

(5) set u = f, and go to step 3; 

else 

(6) determine column index q € arg MinjeN Gp, j; 

(7) if p,q < 0, convert dpq to 1, eliminate the other nonzeros in the column by 
elementary transformations, and go to Step 5. 

(8) compute a& = —Xj,/Ap,n+1) 

(9) update Xp = XB + Aan+1 and f — f= a; 

(10) determine p € argmin {x;,, i= 1,...,m-+ I}. 
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(11) stop if x;, = 0 (optimality achieved); 
(12) set rd = f, and go to step 3. 
end if 


Example 16.4.1 Solve the following problem by Algorithm 16.4.1 : 


min f = 3x1 — 4x2 + 8x34 x4, 


s.t. Xyt2x. +434+5x4+X5 = 3, 
3x, +4x2 — 7x3 — x4 + x6 = 20, 

—2x, —x2  +2x34+2x4 + x7 ==5; 
—x; +x2 —2x3 — x4 + xg =—2; 
—6x1 + 8x2 — 16x3 — 2x4 +x9 = 50, 


xj>0, j=Hl,...,9. 


Answer Initial tableau: 


xX] x2 x3 X4 x5 X6 X7 xg 
1 2 1 ee | 
-3 4 =7 -=l 1 
—2* —1 1 2 1 
-1 1 —2 -l 1 


Step 1 Call Algorithm 14.4.1. 
Iteration 1. 
1. min {3, 20, —5, —2, 50} = —5, p=3. 
3. min {—2, —1,1,2}=-2, g=1 
5. Multiply row 3 by —2, and add —1, 3, 1, 6, —3 times of row 3 to rows 1, 
2, 4, 5, 6, respectively: 


x1 x2 x3 X4 X5 X6 x7 XE 
3/2 3/2 6 1 1/2 
11/2. -17/2 —4 1 —3/2 
1 17. 172: -=1 =1/2 
372 2 -1/2 1 
11 -19 -8 —3 
-11/2 19/2 4 3/2 


The primal feasibility is attained. 


Step 1(2) return with uw = f = 15/2. 
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Iteration 2. 
Transform the preceding tableau into the reduced simplex tableau by taking p = 


6,g =2. 
Change the end entry of the bottom line to the value, 0, of x2. Multiply the bottom 


row by —11/2, and add —3/2, —11/2, —1/2, —3/2, —11 times of row 6 to rows 1- 
5, respectively: 
X19 x2 X3 x4 X5 1X6 x7 xg x | ff fH15/2 
45/11 78/111 10/11 / 3/11 | 1/2 
1 0 1 0 -1 55/20 
1 4/ll 7/11 —4/11 | -1/11 5/2 
1/11 —10/11 -1/ll 1 3/11 1/2 
0 0 0 1 2 65 
1 —19/11 —8/11 —3/11 | 0 


Step 2 Call Algorithm 16.2.3 to seek the dual feasibility. 
1. min{—3/11, —1, —1/11, —3/11, —2,2/11} = —2 <0, p=5. 
3. a = —(65)/(—2) = 65/2. 
4. Xp = Xp + (65/2)ajo9 = (—92/11, —5, —5/11, —92/11, 0, 65/11), 
f = 15/2 — 65/2 = —25; 
5. min {0, 0, 0} > 0; 
Step 2(2) Return with A = —25. 
Step 3 ra = (—25 + 15/2)/2 = —35/2, a = —25 — (—35/2) = —15/2. 
Step 4 xg = Xgp+(—15/2)a19 = (—139/22, 5/2, 5/22, —139/22, 15, 50/11)", 
f = —35/2. 
Step 5 min{—139/22, 5/2,5/22, —139/22, 15, 50/11} = —139/22 < 0, p= 
1. 
Step 6(6) min{45/11, 78/11, 10/11} = 10/11 > 0, gq =7. 
(8) a = —(—139/22)/(—3/11) = —139/6; 
(9) xg = xp + (—139/6)a10 = (0,77/3, 7/3, 0, 184/3, 1/3)1, f 
—35/2 + 139/6 = 17/3; 
(10) min{0, 77/3, 7/3, 0, 184/3, 1/3} = 0; 
(11) the optimal basic solution and optimal value are: 


x* = (7/3, 1/3, 0, 0, 0, 77/3, 0, 0, 184/3)', f* = 17/3. 


The revised version of Algorithms 16.4.1 is omitted. 
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16.5 Notes on Reduced Simplex Algorithm 


We make the following remarks on the reduced simplex algorithm: 

First of all, it is the first simplex algorithm that searches along the objective- 
edge direction. In contrast to the simplex algorithm, it determines the pivot row first 
and the pivot column after. Moreover, it determines its pivot without selection in 
essence, compared with the simplex algorithm that selects the pivot from multiple 
candidates, which is unprecedented. 

Secondly, while its computational effort per iteration is the same as the simplex 
algorithm, the reduced simplex algorithm has much better numerical stability than 
the former. In fact, it tends to determine a large pivot in magnitude, compared with 
the former that would select a pivot arbitrarily close to zero. 

In addition, it is worth mentioning that the generalization of the reduced simplex 
algorithm turns out to be a desirable framework for the controlled-branch ILP solver 
(Part I), which would have potential significance for its applications. 

The birth of the reduce simplex algorithm has been very exciting to the author. In 
the first edition of this book, the original derivation of its prototype was introduced 
in the chapter same-named and was then transformed into the “improved” version in 
the subsequent chapter. When preparing the second edition of this book, the review 
of the bisection simplex algorithm inspired us to find the concise derivation, so we 
can introduce it in a single chapter. 

As early as 30 years ago, in the paper entitled “A Simplex-like algorithm with 
bisection for linear programming” (p. 724), the author indicated: 


(Actually, the proposed is) an approach to improving feasible solutions with the dual type 
of canonical form; in a manner similar to that in the classical (simplex) algorithm. 


We are now aware of the so-called dual type of canonical form being just the reduced 

simplex tableau. At that time, we seemed to be only one step away from the current 

version of the reduced simplex algorithm, but this step has been taken for 30 years! 
In the last paragraph of the paper (p. 742), the author wrote: 


Finally, as the classical (simplex method), the new method is also simplex-like, a purely 
linear one. Because the problems encountered are linear, I feel it is more natural to pursue 
an ‘ideal’ linear method, and it might be wise not to ignore this line while pursuing another 
(nonlinear) method for linear programming. 


Obviously, the preceding was aimed at the upsurge of the interior-point algorithm 
research at that time. A brief dip into the past 30 years serves to confirm this view. 
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Chapter 17 ® 
D-Reduced Simplex Method od 


Consider the following special form of the standard LP problem: 


min c!x, 

st. Ax=e, x>0, a) 
which is termed D-reduced problem. Its right-hand side is a unit vector, and hence 
the dual problem of it is an analogue of the reduced simplex problem. Therefore, it 
should be advantageous to deal with this problem from the dual side. 

If a standard LP problem has a nonzero right-hand term b, 4 0, the problem can 
be transformed into a D-reduced problem by multiplying the rth equality by 1/b,, 
and adding an appropriate multiple of it to the other equalities. 

There are usually multiple choices of row index r. A too small b, is clearly not 
appropriate. Considering numerical stability, it seems to be favorable to take the 
largest in magnitude. On the other hand, a row involving the minimum number of 
nonzeros should be chosen to reduce fill-ins. Numerical stability and sparsity should 
be balanced in large-scale sparse computations. 

In this chapter, we introduce the D-reduced simplex tableau in the next section 
and describe the dual D-reduced simplex algorithm with the Phase-I procedure in 
the subsequent section. In the last two sections, we present the D-reduced simplex 
algorithm with the Phase-I procedure and the bisection dual D-reduced simplex 
algorithm, respectively. 

As it plays a specially role in the new method, the r-indexed row is separated off 
to handle. To this end, introduce the following column and row index sets: 


B={j,.--,jm-1} N=A\B, R={l,...,m}\{r} = {i1,..-,im—i}- 
(17.2) 
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17.1 D-Reduced Simplex Tableau 


Let 1 € R°"—)*(™—D pe the unit matrix. Assume that the initial tableau of problem 
(17.1) is converted to the following so-called D-reduced (simplex) tableau by the 
Gaussian-Jordan elimination with row and column exchanges: 


xp xy  f | RHS 
T Nr 


17. 
Or, 1 at) 
zt 1) -f 


where the mth (r-indexed) row is termed datum row. Of course, a D-reduced 
tableau represents the original problem itself, just as a conventional simplex tableau 
although now the RHS column alone does not give a basic solution. 

It is noted that the datum row represents the equation 


@yxn = 1, (17.4) 
whereas the bottom (objective) row represents equation 
a7 pa Fa 
Zytn—f=a—f. (17.5) 


Lemma 17.1.1 [f the nonbasic entries in the datum row are no more than zero, 
there is no feasible solution. 


Proof Since ay < 0, if x > 0 is a feasible solution, then it follows that the left side 
of (17.4) is less than or equal to 0, as contradicts the right-hand side 1. Therefore, 
there is no feasible solution. oO 


Denote by a,(R) the subvector of dg, indexed by set R. 
Definition 17.1.1 If it holds that 


Oy #0, % =0, (17.6) 


the gth column is a datum column of the D-reduced tableau (17.3). 


Proposition 17.1.1 The D-reduced tableau with the qth datum column gives the 
basic solution 


ip=—XgGg(R), Iy=1/oy, =0, G#JEN, (17.7) 


with objective value f. 
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Proof The D-reduced tableau (17.3) corresponds to problem 


max f = f +21 xn, 
s.t. xp + Nrexy = 0, (17.8) 
olan =, xB, Xn > 0. 


Substituting x; = 0, gq # j € N, Xq = 1/@zg into the equality constraint gives 
(17.7). Carrying out elementary transformations, by taking @, as pivot, will convert 
(17.3) to a conventional simplex tableau, associated with a basic solution, which is 
just that given by (17.7). If 7 = 0, the bottom row remains unchanged, and hence 
the according objective value of the solution is equal to f. Oo 


It is seen that a D-reduced tableau could give multiple basic solutions because its 
datum column can be specified differently. 
The D-reduced tableau satisfying 


2g =90, @ >90; Gig <9, IER, (17.9) 
is called feasible, whereas satisfying 
Zn = 0, (17.10) 


called dual feasible. 


Lemma 17.1.2 A feasible D-reduced tableau corresponds to a feasible basic 
solution. A dual feasible D-reduced tableau satisfying (17.6) corresponds to a dual 
feasible basic solution. 


Proof Consider the classical simplex tableau, resulting from the basis change by 
taking @, # 0 as pivot. If (17.9) holds, the basic solution x > 0, defined by (17.7), 
is clearly a feasible solution. If (17.6) and 12.12b both hold, the tableau corresponds 
to a dual basic feasible solution zz = 0, Zy > 0. oO 


Lemma 17.1.3 A D-reduced tableau satisfying 


@q>0, Gig <0, ie€R, (17.11) 
Zy=0, inv >0, (17.12) 


corresponds to a pair of primal and dual optimal basic solutions. 


Proof Under the assumption, conditions (17.9) and (17.10) are both satisfied. By 
Lemma 17.1.2, the D-reduced tableau corresponds to a primal and a dual feasible 
basic solution which are complementary. Oo 
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Lemma 17.1.4 Let (17.3) be a feasible D-reduced tableau. If, for some q' € N, it 
holds that 


2q! < 0, Gi, q! <Q, 7 ER, @q! <0, 


then the problem is lower unbounded. 


Proof The feasibility of the D-reduced tableau implies satisfaction of (17.9) for 
some q € N. Carrying out elementary transformations, with @, > 0 as the pivot, 
will lead to a feasible simplex tableau with the sign of the g’ column remaining 
unchanged; there is no change to the objective row, and hence to Z, < 0. Therefore 
the problem is lower unbounded (Lemma 3.1.2). oO 


Lemma 17.1.5 [fa D-reduced tableau satisfies 
Zqg <0: Gig <0, ieR, (17.13) 


then the problem has no feasible solution or has no dual feasible solution with 
objective value f. 


Proof Consider the dual problem, represented by D-reduced tableau (17.3), i.e., 


max f + g, 


I 
s.t. (ire JC)+ By ( OY, ze, we0, yer! ger, 
Np On g ZN ZN 


(17.14) 


where the objective function involves a single variable g. Setting g = O turns this 
problem to 


I ZB 0 (17.15) 
s.t. - + =(_ |, ‘ > 0, 
(iin) (:") (2) aie 


where the objective value equals f. On the other hand, deleting the mth row of 
the tableau gives a classical simplex tableau of (17.15). Since (17.13) holds, by 
Lemma 15.4.1 the problem either has a feasible solution or is lower unbounded. If 
there is no feasible solution, the D-reduced tableau has no feasible solution: if lower 
unbounded, then (17.15) has no feasible solution, hence there is no dual feasible 
solution, associated with objective value f. oO 


It will be seen a little later that the datum row corresponds to the dual reduced 
gradient, and the basis change, carried out on the D-reduced tableau, only touch the 
R-indexed rows, except the objective value f. Desired changes in objective value 
will be realized by adding appropriate times of the datum row to the bottom row. 
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It is clear that an initial D-reduced tableau may be obtained from a (classical) 
simplex tableau as follows. If b, of the right-hand side is nonzero, multiplying the r- 
indexed row by 1/b,, and then adding appropriate times of the row to the other rows 
respectively gives a D-reduced tableau with the r-indexed row being datum row. 
The j,-indexed column can be taken as the datum column, as it becomes nonbasic, 
ie., 


ay, jp = Abe Be jy = —b;j/b,-, i € R, 


corresponding to zero reduced cost. Further, if the conventional tableau is feasible, 
i.e., b > O, then the resulting is a feasible D-reduced tableau. 


17.2. Dual D-Reduced Simplex Algorithm 


Starting from a dual feasible D-reduced tableau, the method presented in this section 
generates a sequence of such tableaus to achieve primal feasibility until achieving a 
pair of primal and dual optimal basic solutions. 

Let (17.3) be such a tableau and let (17.10) hold. It is noted by Preposition 17.1.1 
that the objective value of the basic solution x is f, which is equal to the opposite 
number of the south-eastern entry of the tableau. 

If entries in the datum row are all less than or equal to 0, the problem has 
no feasible solution (Lemma 17.1.1). This is usually not the case, and hence the 
following rule is well-defined. 


Rule 17.2.1 (Dual Column Rule) Determine the column index q such that 
B = Zq/@q = min{Z;/@; |@; > 0, j ¢ N} = 0, (17.16) 


where £ is termed dual stepsize. 
If 6 4 0, add —£ times of the datum row to the bottom row to convert Z, to 0, 
then apply the following row rule. 


Rule 17.2.2 (Dual Row Rule) Select the row index 
p € arg max Gig. 


Since @g > 0, an optimal D-reduced tableau is already obtained if dpg < 0 
(by taking column g as datum column); otherwise, convert it to 1, and eliminate 
the other nonzeros of column q by elementary transformations. Thus, an iteration is 
complete; consequently, column gq enters, and column j, leaves the basis. It is noted 
that the latter becomes the new datum column. 

The resulting tableau remains dual feasible. In fact, Rule 17.2.2 ensures a largest 
possible stepsize 6 subject to dual constraints. Now the south-eastern entry of the 
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tableau becomes — f + B, so the increment of the objective value is just equal to 
B = 0. If B vanishes, the objective value will remain unchanged. 

If it holds that zy > 0, the D-reduced tableau is said to be dual nondegenerate. 
In this case, B given by (17.16) is positive, and hence the objective value strictly 
increases. 

We repeat the preceding steps until optimality is achieved or infeasibility 
detected. It is not difficult to show that, the leaving column j, will not enter the 
basis immediately, and the entering column g will not leave immediately either. The 
basis change will not touch the objective value; only adding a multiple of the datum 
row to the bottom row changes it. Since the f column remains unchanged in the 
process, it can be omitted from the tableau. 

The overall steps can be put into the following model. 


Algorithm 17.2.1 (Dual D-Reduced Simplex Algorithm: Tableau Form) Initial: 
dual feasible D-reduced tableau of form (17.3). This algorithm solves the D-reduced 
problem (17.1). 


. Stopif J ={j eN| aj > O}=%. 

. Determine # and column index q such that B = Z,/@g = minjey Zj/@;. 

. If 6 > 0, add —£ times of datum row to the bottom row. 

. Determine row index p € arg maxjeR Gj q- 

. Stop with x7 = 1/@g, XB = —Xqq(R) if dpg < 0. 

. Convert dp q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

7. Go to step 1. 


NnBWN Ke 


Theorem 17.2.1 Under the dual nondegeneracy assumption, Algorithm 17.2.1 
terminates either at 


(i) Step 1, detecting infeasibility of the problem, or at 
(ii) Step 5, giving a basic optimal solution. 


Proof The proof of the termination under the dual nondegeneracy is the same as 
in the classical simplex context. Meanings of the exits come from Lemmas 17.1.1 
and 17.1.3, as well as discussion preceding the Algorithm Oo 


This Algorithm can be viewed as solving the dual problem through the tableau. 
To explain, return to the dual problem (17.14). 
It is clear that there is the following solution to the dual problem: 


zp=0, zn =2nN, 


which is feasible when zy > 0. The reduced gradient of the objective function is 
given by 
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along which the objective value in the reduced space increases the fastest. The 
associated vector in z space is then 


Azgp=0, Azy =—oan. 
This leads to the following update: 
2n=Zn—Bon, Z=B. (17.17) 
It is noted that the dual stepsize 6 > O equals the according increment of 
the objective value. If my <¢ O, the greatest possible value, maintaining dual 
feasibility, is 
B = 2q/@q = min{Z;/@; |o; > 0, j € N}. (17.18) 


It is then seen that the dual D-reduced simplex algorithm actually deals with the 
dual problem according to the idea of the reduced simplex algorithm. 


Example 17.2.1 Solve the following problem by Algorithm 17.2.1: 


min 3x; +5x2 + 2x3 + 4x4, 


s.t. 2x1 + 5x2 — 3x3 — 4x4 + x5 = —2, 
—4x, — 3x2 + 6x3 — 2x4 + x6 = 4, 
—x, — 4x2 — x3 + 2x4 + x7=—5, 


xj>0, j=l,...,7. 


Answer Initial tableau 


nal x2 X3 x4 X5 x6 47 RHS 
2 5 3 —4 1 —2 

_4 3 6 -2 1 4 

-1 —4 -1 2, 1 —5* 
3 5 2 4 


Iteration |: 
max{| — 2], |4],| — 5|} = 5, r = 3, take the third row as datum row. Multiply row 
3 by —1/5, and add 2, —4 times of row 3 to rows 1,2, respectively, resulting in the 
following D-reduced tableau: 

which is dual feasible, as the bottom line is nonnegative. 

Call Algorithm 17.2.1. 


Iteration 2: 


1. J ={1,2,3} 49. 
2. B = min{3/(1/5), 5/(4/5), 2/(1/5)} = 5/(4/5), g = 2. 
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RHS 


RHS 


440 
XxX x2 X3 X4 x5 X6 x7 
12/5 33/5* —13/5 24/5 1 -2/5 
-24/5 31/5 26/5 2/5 1 4/5 
1/5 4/5 1/5 -2/5 1/5 
3 =) 2 4 
3. Add —5/(4/5) times of row 3 to the bottom row. 
4. max{33/5, —31/5} = 33/5 > 0, p= 1. 
6. Multiply row | by 5/33, and add 31/5, —4/5 times of row | to rows 2 and 3, 
respectively: 
x] x2 x3 x4 X5 x6 
4/111 13/33 8/11 5/33 
—28/11 91/33* —S4/11 31/33 1 
-1/1l 17/33 2/11 —4/33 
7/4 3/4 13/2 
Iteration 3: 
-S=3, 4540. 


NBWNr 


91/33} = 91/33 > 0, p=2. 


3, respectively: 


249/13 
5/13 
32/17 


Iteration 4: 


nABWN 


x2 x3 x4 x5 X6 x7 
i —10/7 2/7 1/7 

1 -162/91 31/91 = 33/91_—Ss2/13 

100/91 —27/91  —-17/91  —3/13 

106/17 3/17 25/17 


J= {1,44 9@. 
. B = min{(32/17)/(5/13), 106/17)/(100/91)} = B2/17)/(5/13), ¢g = 1. 
. Add (32/17)/ 
. max{—12/13, 
. X; = 13/5. The optimal basic solution and optimal value: 


(5/13) times of row 3 to the bottom row: 
0} < 0. 


- B = min{(3/4)/17/33), 3/2)/(2/11)} = 3/4)/0.7/33), ¢ = 3. 
. Add —(3/4)/(17/33) times of row 3 to the bottom row. 
. max{—13/33, 
. Multiply row 2 by 33/91, and add 13/33, —17/33 times of row 2 to rows | and 


RHS 


—131/17 


x* = (13/5, 0, 12/5,0,0,0,0)',  f* = 63/5. 
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xX] x2 1X3 X4 x5 X6 x7 RHS 
1 —10/7 2/7 1/7 
—12/13 1 —162/91 31/91 33/91 2/13 
5/13 100/91 —27/91  —17/91  —3/13 1 
6/7 57/35 32/35 13/5 —63/5 


Further, we give the revised version of Algorithm 17.2.1. Denote by Br the 
submatrix consisting of entries associated with B, R (similarly below). Assume that 
row r = mis the datum row. It is not difficult to derive the following revised tableau 
which is equivalent to the D-reduced tableau (17.3): 


ra te f | RHS 
: Bu Na 17.19 
elN —e! BBR'Nr 1 ie 


T T p—! 
Cy —CpBp Nr |-1 


Assume the column g enters the basis. Introduce ag(R) = Br dg (R), where 
dg(R) is ag’s subvector associated with row index set R. The formula for updating 
the inverse of the basis matrix is 


B,| = EsBz', (17.20) 
where 
1 —4i; q/Gi,,q 
—i,_1q/Ap,q 
E,= 1/di,, 4 (17.21) 


Gis 4) q/ Gig, q 


—Gin_1,q/ Gis, q 1 


Based on equivalence between (17.3) and (17.19), a revised version of Algo- 
rithm 17.2.1 is stated as follows. 


Algorithm 17.2.2 (Dual D-Reduced Simplex Algorithm) Initial: B,R,N 
defined by (17.2). BR: and zy > 0. This algorithm solves the D-reduced problem 
(17.1). 


1. Compute ay = Nlen — NEBR! Bem. 
2. Stop if J = {j € N | @; > 0} = G (dual unbounded problem). 
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. Determine 6 and column index q such that B = Z,/@g = minjey Zj/@j;. 
. If B > O, update by Zy = Zy — Bon. 

. Compute Gy (R) = Ba'ag(R). 

. Select p € argmaxjer Gig (p = is). 

. Ifdp,q < 0, compute 


NAYDHAN BW 


- a= on me, as 7 = T- 
Xg=1/@g, Xp=—Xgaqg(R), f =cgXq + Cprs.- 


8. Stop (optimality achieved). 

9. Update Bz’ by (17.20). 
10. Update (B, N) by exchanging j, and q. 
11. Go to step 1. 


17.2.1 Dual D-Reduced Phase-I 


The dual D-reduced algorithm has to start from a dual feasible tableau/solution. If 
ZN =C n-Np Be B = 0, sucha starting point is readily available; otherwise, there 
is a need for dual Phase-I procedure to provide it. An initial dual feasible tableau 
for Algorithm 17.2.1 can be obtained by the following modification of D-reduced 
simplex Algorithm 17.3.1. 


Algorithm 17.2.1 (Dual D-Reduced Phase-I: Tableau Form) Initial : D-reduced 
tableau of form (17.3). This algorithm finds a dual feasible D-reduced tableau. 

1, Determine column index q € arg Min jen Z;. 

2. Stop if Zq = 0. 

3. If @g #0, add B = —Zq/Wg times of the datum row to the bottom row. 

4. Determine row index p € arg MaXxjeR Qj q. 

5. Go to 7 if dpq > 9. 

6. Stop if @q = 0. 

7. Convert Gpq to 1, and eliminate the other nonzeros in the column by 
elementary transformations. 

&. Go to step 1. 


Theorem 17.2.2 Assume termination of Algorithm 17.2.1. It terminates either at 


(i) step 2, giving a dual feasible D-reduced tableau, or at 
(ii) step 6, detecting infeasibility or lower unboundedness. 


Proof Validity of the meaning of exit 2 is clear. Assume that it returns from exit 6. 
Then Z7 < 0,ag < 0 and @, = 0 hold. If the problem is feasible, by Lemma 17.1.1 
there is g’ € N such that @, > 0. Taking @,’ as pivot, a (conventional) simplex 
tableau is obtained by carrying out elementary transformations, without changing 
the g column. As components of the column are positive, the problem is infeasible 
or lower unbounded, by Lemma 15.4.1. oO 
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Example 17.2.2. Solve following problem using Algorithm 17.2.1 as Phase-I: 


min —3x; — 5x2 + 4x3 + 8x4, 


S.t. 6x1 — 2x2 + 3x3 + 2x4 + x5 = 2: 
—2x, +4x. — x3 -— 8x4 + x6 = —4, 
4x, +5x2+2x3 +24 +x7= 6, 


xj>0, j=l,...,7. 


Answer Initial tableau 


Iteration 1: 
Arbitrarily take row 2 to be as the datum row. Multiply the row by —1/4, and add 
—2, —6 times of it to rows 1,3, respectively: 


xX] x2 X3 X4 x5 X6 
5 5/2 —2 1 1/2 
1/2 -1 1/4 2; —1/4 
1 11* 1/2 -l1 3/2 

oe a 8 


Dual Phase-I: call Algorithm 17.2.1.7r = 2. 
Iteration 2: 


. min{—3, —5, 4, 8} = —5 <0, g =2. 

. Add B = —(—5)/(—1) = —5 times of row 2 to the bottom row. 
. max{0, 11} = 11>0, p=3. 

. Add 1/11 times of row 3 to the bottom row: 


NI BWR 


xjle x2 x3 X4 1X5 X6 x7 
5* 5/2 —2 1 1/2 
13/22 13/44 1 —5/44 1/11 
1/11 1 1/22 -l 3/22 1/11 


=i riya 3 5/4 
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Iteration 3: 


. min{—11/2, 11/4, —2,5/4, 0} = -11/2 <0, q=1. 

. Add 6B = —(—11/2)/(13/22) times of row 2 to the bottom row. 

. max{5,1/l1l}=5>0, p=1. 

. Multiply row 1 by 1/5, and add —13/22, —1/11 times of row 1 to rows 2 and 3, 
respectively: 


NWR 


xX} x2 x3 X4 x5 X6 x7 RHS 
1 1/2 —2/5 1/5 1/10 
68/55 —13/110 —19/110 1/11 1 
1 —53/55 —1/55 7/55 1/11 
11/2 95/13 5/26 11/13 | 56/13 


Iteration 4: 


1. min{11/2, 95/13, 0, 5/26, 11/13} > 0, g =1. 
2. Dual feasibility is achieved. 


Dual Phase-II: call Algorithm 17.2.1:r = 2, q =2. 
Iteration 5: 


1. J = {4, 7}. 

2. B = max{—(95/13)/(68/55), —(1/13)/(./11)} = —(95/13)/(68/55), q = 
4. 

3. Add £6 times of row 2 to the bottom row. 


Xj x2 x3 x4 X5 X6 x7 RHS 
1 1/2 —2/5 1/5 1/10 
68/55 —13/110 —19/110 1/11 1 
1 —53/55 —1/55 7/55 1/11 
11/2 95/136 165/136 21/68 | —109/68 


4. max{—2/5, —53/55} < 0. 
5. Take x4 = 55/68. The optimal basic solution and optimal value: 


x* = (11/34, 53/68, 0, 55/68,0,0,0,1)', —f* = 109/68. 


Based on the equivalence between the reduced tableau (17.3) and the revised tableau 
(17.19), the tableau Algorithm 17.2.1 can be turned to the following revision. 


Algorithm 17.2.2 (Dual D-Reduced Phase-IT) Jnitial: B,R,N defined by 
(17.2). Be: ZN =CN— Nt Brice. This algorithm finds a dual feasible D-reduced 
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basis. 


Determine column index q € arg miN jen Z;- 

. Stop if Zq = 0 (dual feasibility achieved). 

. Compute ON = Nl em — Ni(Ba Ben): 

If @q # 0, update zn = Zy + Bon, where B = —Zq/aq. 
. compute ag(R) = By dq(R). 

. Determine p € argMaxjeR Aig (Pp = is). 

Go to 9 ifap,q > 9. 

. Stop if @q = 0 (infeasible or lower unbounded). 
. Update Bz" by (17.20). 

. Update (B, N) by exchanging j,; and q. 

. Go to step 1. 


SSO PNDAWAWNN 


BS NY 


There have been no numerical results with the algorithms, described in this 
section. It seems to be better to obtain a (conventional) dual feasible simplex tableau 
(basis) first, and then convert it to a D-reduced tableau (basis). 


17.3. D-Reduced Simplex Algorithm 


Starting from a feasible D-reduced tableau, the so-called D-reduced simplex 
algorithm, described in this section, proceeds toward dual feasibility, iteration by 
iteration, while maintaining primal feasibility. 

Let (17.3) be a feasible D-reduced tableau with column gq as datum column, 
satisfying conditions (17.6) and (17.9). Column pivoting is carried out first. 


Rule 17.3.1 (Column Rule) Select the column index 
q € argmin{z;| j ¢ N, j #4}. (17.22) 
If Zy = 0, it is clear that optimality is already achieved. Now assume that 
Zq <0. (17.23) 
According to whether 
digi $0, ieER (17.24) 


holds or not, one of the following two cases is handled: 


(i) Condition (17.24) holds. 
If @g < 0, the problem is lower unbounded (Lemma 17.1.4). If @g > 0, 
then set g = g’ and compute 


BS = 9/84 (17.25) 
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and add £ times of the datum row to the bottom row, so that z, is zeroed. Taking 
column q as the new datum column, it is clear that the D-reduced tableau is 
still feasible, thus an iteration is complete. As 6 > 0, the objective value f, 
associated with the tableau, strictly decreases. 

(ii) Condition (17.24) does not hold. 
Then the following rule is well-defined: 


Rule 17.3.2 (Row Rule) Determine the row index p such that 
@ = —Apq/Apq' = Min{—jg/Aig' | dig) > 0, i € R}. (17.26) 
It is known from (17.9) and (17.26) that 
Gpq <9, apg > 9. (17.27) 


Therefore, «@ > 0. When the strict inequality in (17.9) holds, the D-reduced tableau 
is said to be nondegenerate. In this case, a > 0. 


Lemma 17.3.1 Assume that (17.23), (17.6), and (17.9) hold, and q’ and p 
are determined by (17.22) and (17.26), respectively. Then the problem is lower 
unbounded if 


Wy <0, a> —Gg/dq'. (17.28) 
Proof Multiply the datum row of the D-reduced tableau by 1/@,, then fori € R, 
add —adjq times of the datum row to the ith row. The resulting is a (conventional) 
feasible simplex tableau. The q’ column’s component at the datum row of this 
tableau is 


ig = Gy Bq < 0, (17.29) 


where the inequality comes from the first expression of (17.6) and of (17.28). 
Besides, the other components of g’ column are 


Gig! = Gig! — Gig(@q'/@q), LER. (17.30) 
If dig: < 0, i € R, then from (17.9), (17.29), and (17.30), it is known that aj,’ < 0; 
and if ajg: > 0, i € R, from the second expression of (17.28) and (17.26), it follows 
that 


— Gig [aig = —@q/Gqy >0, ie R, (17.31) 


where the strict inequality is from the first expression of (17.6) and of (17.28). From 
Gig! > O and (17.28), it follows that aj, < 0 and 


—Giq' /Aig < —Wq'/Wq. 
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Multiplying the preceding by —aj, > 0 and combining the resulting expression 
with (17.30) lead to ag’ < 0. Thus, it was shown that the q’ column of the feasible 
simplex tableau is less than or equal to zero. By Lemma 3.1.2, the problem is lower 
unbounded. Oo 


Now assume that (17.28) does not hold. Execute elementary transformations on 
the feasible D-reduced tableau to convert dp 4 to 1, and eliminate the other nonzeros 
in the column (including those in the bottom row). Thus, xj, leaves and x,’ enters 


the basis. As p € R, the objective value f remains unchanged. If the q-indexed 
entry of the resulting bottom row is nonzero, add appropriate times of the datum 
row to the bottom row to eliminate it. For the resulting tableau, the following result 
stands good. 


Proposition 17.3.1 Assume that (17.23), (17.6), and (17.9) hold, and q' and p are 
determined by (17.22) and (17.26), respectively. If @q = 0, or 


Gq <0, & <—Gq/aq, (17.32) 


then the new tableau is a feasible D-reduced tableau, with column q as the datum 
column. The according objective value never increases and strictly decreases under 
nondegeneracy assumption. 


Proof Mark entries of the new tableau by e. The following inequality will be shown 
first: 


Gig <9, TER. (17.33) 
The p (4 r)th component of the g column of the new tableau is 
Gpg = Gpq/Apq' < 9, (17.34) 


where the inequality follows from (17.9) and the first expression of (17.27). The 
other components of the column are 


dig = Gig — Gig'(Gpq/Gpq'), LER, LAD. (17.35) 
Fori € R,i 4 p, if Gig! < 0, then by (17.9), (17.34), and (17.35), it is known that 
Gig < 0; and if Gig! > 0, by (17.26) and (17.35), it is known that aig < 0. Therefore, 
(17.33) hold. 


On the other hand, the g column’s component at the datum row of the new 
tableau is 


Og = Wg — Wg! (Apg/Apq’). (17.36) 
If @g: = 0, from the first expression of (17.6), (17.34) and (17.36), it follows that 


dg > 0. (17.37) 
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If, otherwise, (17.32) holds, from the second expression of (17.6) and (17.26), it 
follows that 


—Gpq/Apgq' < —Wq/q'. 
Multiplying the preceding by @,' < 0 gives 
—@q' Apq/Apq') > —&q; 


combining which and (17.36) leads to (17.37). 
Note that z, = 0. After a basis change by taking dpq’ as pivot, the bottom row’s 
entry at the g column equals 


v= Za <0, 


where the inequality follows from (17.26) and (17.34). If v = 0, then it is 
known by (17.33) and (17.37) that resulting is a feasible D-reduced tableau with 
q column as the datum one, and the according objective value does not increase. If 
nondegeneracy is assumed, then w > 0 and v < 0. Add —v/@, times of the datum 
row of the current tableau to the bottom row to eliminate its entry at the g column, 
the right-hand side entry at the bottom row becomes 


—(f + v/dq). 


Thus, from v < 0 and @, > 0, it is known that the associate objective value strictly 
decreases. Oo 


The preceding steps are repeated until an optimal tableau is obtained or the lower 
unboundedness of the problems detected. The overall steps are put into the following 
algorithm. 


Algorithm 17.3.1 (D-Reduced Simplex Algorithm: Tableau Form) Initial: fea- 
sible D-reduced tableau of form (17.3) with g-indexed column as the datum one. 
This algorithm solves D-reduced problem (17.1). 


1. Select column index q’ € argmin{z; | j ¢ N, j # q}. 

2. Stop with xg = 1/@g, XB = —Xgdq(R) if Zq = O. 

3. Goto step6if 7 = {i € R| ajg > O} FG. 

4. Stop if @g < 0. 

5. Set g = q', and go to step 9. 

6. Determine @ and row index p such that a = —@pq/@p q’'= min{—Gj q/Gj q’ | Gq! 
>0, i € R}. 


7. Stop if @g < Oanda = —@q/@q'. 
8. Convert dp q’ to | and eliminate the other nonzeros in the column by elementary 
transformations. 
9. Add B = —Z,/@z, times of the datum row to the bottom row. 
10. Go to step 1. 
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Theorem 17.3.2 Under nondegeneracy assumption, Algorithm 17.3.1 terminates 
either at 


(i) Step 2, giving a basic optimal solution, or at 
(ii) Step 4 or 7, detecting lower unboundedness of the problem. 


Proof The proof of the termination of the Algorithm under the assumption of 
nondegeneracy is the same as that in the conventional context. Meanings of its exits 
follow from Lemmas 17.1.3, 17.1.4, and 17.3.1, as well as the discussions preceding 
the Algorithm. oO 


Example 17.3.1 Solve the following problem by Algorithm 17.3.1: 


min 5x; — 4x2 + 7x3 — x4, 


s.t. —3x, — 2x2 + 5x3 — 4x4 + x5 = 4, 
—6x, — 4x2 + 3x3 4+ 2x4 + x6 = 2, 
xy +5x2-—4x3+ 1x4 + x7 =3, 


xj20, jol,...,7. 


Answer Initial tableau 


Iteration 1: 
max{|4], |2|, |3|} = 4, r = 1. Take the first row as pivot row. Multiply the row by 
1/4, and add —2, —3 times of it to rows 2, 3, respectively: 


Xi} X2 X3 X4 X5 X6 
—3/4 —1/2 5/4 1 1/4 
-9/2 -3 1/2 4 -1/2 1 
13/4 13/2* —31/4 4 —3/4 
5 —4 7 -1 
x5 becomes a nonbasis variable. For the tableau, aj,5 = 1/4 > 0, aa5 = 


—1/2 <0, a3,5 = —3/4 < 0, hence it is a feasible D-reduced tableau. 
Call Algorithm 17.3.1.r=1, q=5. 
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Iteration 2: 


min{5, —4,7, -l} = —4 <0, q’ =2. 

.1= 3) 49%. 

a = min{—(—3/4)/(13/2)} = 3/26, p = 3. 

a2 = —1/2 < 0, 3/26 < —(1/4)/(—1/2) = 1/2. 

. Multiply row 3 by 2/13, and add 1/2,3,4 times of row 3 to rows 1, 2, 4, 
respectively: 


IAW 


xX, x2 x3 X4 X5 1X6 x7 RHS 
—1/2 17/26 = —9/13 5/26 1/13 1 
=3 —40/13. 76/13 11/131 6/13 

1/2 1 31/26 8/13 —3/26 2/13 
i 29/13 19/13 —6/13 8/13 


xX) x2 x3 X4 X55 X6 Xx7 RHS 
-1/2 17/26 —9/13 5/26 1/13 1 
—3 —40/13 76/13* —11/13 1 6/13 

1/2 1 —31/26 8/13 —3/26 2/13 
29/5 19/5 =1/5 4/5 12/5 


Iteration 3: 


1. min{29/5, 19/5, —1/5, 4/5} = —1/5 < 0, q’ =4. 
. 1 = {2,33} 49. 
6. a = minf—(—11/13)/(76/13), —(—3/26)/(8/13)} 
= —(—11/13)/(76/13) = 11/76, p = 2. 
7. a1,4 = —9/13 <0, 11/76 < —(5/26)/(—9/13) = 5/18. 
8. Multiply row 2 by 13/76, and add 9/13, —8/13, 1/5 times of row 2 to rows 1, 3, 
and 4, respectively: 


w 


XxX] x2 X3 X4 X5 X6 X7 RHS 
—65/76 11/38 7/76 9/76 5/38 1 
—39/76 —10/19 1 —-11/76 = 13/76 3/38 

31/38 1 33/38 S138. covyis: - D9 
433/76 351/95 —11/380 13/380 31/38 | 12/5 
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9. Add B = —(—11/380)/(7/76) times of row | to the bottom row. 


XxX] x2 X3 X4 X5 X6 XT RHS 
—65/76 11/38 7/16 9/76 5/38| 1 
—39/76 —10/19 1 —11/76 13/76 = 3/38 | 
31/38 1 33/38 -1/38 —2/19 2/19 | 
38/7 53/14 1/14 6/7 | 19/7 


Iteration 4: 


1. min{38/7, 53/14, 1/14, 6/7} = 0. 
2. Set x5 = 76/7. The optimal basic solution and optimal value: 


x* = (0,2/7, 0, 11/7, 76/7,0,0,1)7, f* =—19/7. 


A revised version of Algorithm 17.3.1 can be obtained by deriving the search 
direction using the inverse of the basis matrix, as done for the derivation of the 
(conventional ) revised simplex algorithm. But, for simplicity, here we directly apply 
the equivalence between the D-reduced tableau (17.3) and revised tableau (17.19). 

Denote a vector e’s subvector, corresponding to the row index set R, by e(R). 
Then the revised version of Algorithm 17.3.1 can be described as follows. 


Algorithm 17.3.2 (D-Reduced Simplex Algorithm) Initial: B, R, N defined by 
(17.2). Ba’; In = en — NEBR cr, Gy = Nem — N3(Bq Bem). g(R) = 
Ba, (R) < 0, @ > 0, Z%q = 0. This algorithm solves the D-reduced problem 
(17.1). 


1. Determine column index q’ € argmin{z; | j ¢ N, j # q}. 
2. If Zq = 0, compute 


= = os = = a = T- 
Xqg =1/@q, XB = —Xqdq(R), f =CqXq + CpxB- 


3. Stop (optimality achieved). 

4. Compute G(R) = Br'aq/(R). 

5. Goto step7ifl ={i ER | aig > O} FV. 

6. Stop if @g: < 0 (infeasible problem). 

7. Set g = q’, and go to step 12. 

8. Determine @ and p such that @ = —dpq/4pq' = MiNje, —Ajg/Gjq' (P = is). 
9. Stop if @g: < 0 and a = —@q/a@q (unbounded problem). 
10. Compute oy = NEB; ey. 


11. Update oy = Oy + von, @j, = v, where v = —@q/0q. 
12. Update zy = Zy + Bion, Li = Bi, where By = —Zq'/oq. 
13. Update Zy = Zn + Bo@y, where By = —Z,/Oq. 


14. Update Bz’ by (17.20). 
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15. Update (B, N) by exchanging j,; and q. 
16. Go to step 1. 


17.3.1 D-Reduced Phase-I: Most-Obtuse-Angle Rule 


As usual, the D-reduced simplex method, described in the previous section, is only 
a Phase-II procedure. Any conventional Phase-I method can be slightly modified 
to generate a starting point (Chap. 14). Dropping the initial condition zy > 0, in 
addition, the dual Algorithm 17.2.1 can be directly used to achieve primal feasibility. 

According to the analysis made at the second half of Sect. 17.2, the reduced 
gradient of the dual objective function corresponds to the negative datum row in z- 
space. Based on the most-obtuse-angle heuristic, the dual nonnegativity constraint 
(€.g., Zq = 0), whose gradient and the negative datum row form the most obtuse 
angle, should be binding, that is, the associated index be chosen to enter the basis. 
Based on this idea, the following algorithm can be obtained by modifying the dual 
Algorithm 17.2.1. 


Algorithm 17.3.3 (Tableau D-Reduced Phase-I: Most-Obtuse-Angle) Initial: 
D-reduced tableau of form (17.3). This algorithm finds a feasible D-reduced 
tableau. 


. Determine column index q € argmaxjen @;. 

. Stop if @, < 0. 

. Add B = —Zq/@gq times of the datum row to the bottom row. 

. Determine row index p € arg max{djg |i € R}. 

. Stop if dp.g < 0. 

. Convert dp q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

7. Go to step 1. 
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Theorem 17.3.3. Assume termination of Algorithm 17.3.3. It terminates at 


(i) Step 2, detecting infeasibility of the problem, or at 
(ii) Step 5, giving a feasible D-reduced tableau. 


Proof It follows from Lemmas 17.1.1 and 17.2.1. Oo 
Example 17.3.2 Solve the following problem using Algorithm 17.3.3 as Phase-I: 


min f = —x; +5x2 — 3x3 — x4, 

s.t. xy + lx2 + 2x3 + 5x4 + x5 = 3, 
—7x, + 3x2 + 4x3 -— x4 + X6 = —5, 
—2x1 — xo + x3- x4 + x7 = —2, 


—5x1 — 6x2 + 8x3 — 2x4 +xg= 6, 
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Answer Initial tableau 


xX] x2 x3 x4 x5 X6 x7 RHS 

1 1 2 5 1 3* 
-7 3 4 -1 1 —5 
—2 —1 1 —1 1 —2 
=5 —6 8 =2 6 
-1 5 —3 -1 


Iteration 1: 

Arbitrarily take the first row to be as the datum row. Multiply the row by 1/3, 
and add 5; 2; —6 times of it to rows 2, 3,4, respectively, yielding the following 
D-reduced tableau: 


x] x2 x3 X4 x5 X6 x7 
1/3 1/3 2/3 5/3 1/3 
16/3 14/3. 22/3 22/3* 5/31 
=47i 18 773 1/3 2/3 1 
=4 =8 4 ai =) 
=f 5 = =i 1/5 


Phase-I: Call Algorithm 17.3.3. r = 1. 
Iteration 2: 


. max{1/3, 1/3, 2/3, 5/3, 1/3} =5/3 > 0, gq =4. 

. Add B = —(—1)/(5/3) = 3/5 times of row | to the bottom row. 

. max{22/3, 7/3, —12} = 22/3 >0, p=2. 

. Multiply row 2 by 3/22, and add —5/3, —7/3, 12 times of row 2 to rows 1, 3, 4, 
respectively: 


NRW Re 


XxX] BS) X3 X4 XS5 X6 X7 XE 
17/11 —8/11 -1 —1/22,  —5/22 
—8/11 7/1 1 1 5/22 3/22 
4/11* —20/11 3/22, —7/22 1 
—173/11 —4/11 16 8/11 18/11 1 
—4/5 26/5 —13/5 1/5 


Iteration 3: 


1. max{17/11, —8/11, —1, —1/22, —5/22}= 17/11 >0, g=1. 
3. Add B = —(—4/5)/(17/11) times of row | to the bottom row. 
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4. max{—8/11, 4/11, —173/11} =4/11>0,p=3. 
6. Multiply row 3 by 11/4, and add —17/11, 8/11, 173/11 times of row 3 to rows 
1, 2, 4, respectively: 


xX] x2 x3 X4 x5 X6 x7 xg RHS 
7 = =5/8 9/8 —17/4 1 
3 2 12 2 
1 -5 3/8 -7/8 11/4 
-79 16 53/8 —97/8 173/4 1 
82/17 —53/17 3/17 —2/17 19/17 
Iteration 4: 
1. max{7, —1, —1, —5/8, 9/8, —17/4}=7> 0, g =2. 
3. Add 6 = —(82/17)/7 times of row 1 to the bottom row: 
Pal x2 x3 X4 x5 X6 x7 RHS 
7 -1 —5/8 9/8  —17/4 1 
3 1* 1 1/2 —1/2 2 
1 —5 3/8 —7/8 11/4 
~79 16 53/8 —97/8  173/4 
17/7 17/28 —25/28 41/14 3/7 
4. max{—3, —5, —79} < 0. 
5. Feasibility is achieved. 
Phase-II: Call Algorithm 17.3.1.r = 1, q = 2. 
Iteration 5: 
1. min{—17/7, 17/28, —25/28, 41/14} = —17/7 <0, q/ =3. 
3.1 ={2,44 49. 
6.a@ =min{—(—3)/1, —(—79/16)}=3, p =2. 
7. 413 =—-1<0, a=3 < -7/(-1) =7. 
8. Add 1, —6, 17/7 times of row 2 to rows 1,4,5, respectively. 
9. Add —(—51/7)/4 times of row | to the bottom row: 
Pal x2 x3 X4 X5 X6 X7 RHS 
4 1 =1/8 5/8 —9/4 
3 ia =12 2 
1 —5 3/8 —7/8 11/4 
—31 11/8 33/8 45/4 
51/32 —31/32 59/16 
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Iteration 6: 


1. min{17/4, 51/32, —31/32, 59/16} = —31/32 < 0, q’ =6. 
3,.7=90. 
4. 5/8 > 0. 
5. q =6. 
9. Add —(—31/32)/(5/8) times of row | to the bottom row: 
x] X2 4X3 x4 X5 X6 x7 Xg ] RHS 
4 1 -1/8 5/8 —9/4 1 
3 61 1 1/2 -1/2 2 
1 -5 3/8 7/8 11/4 
—31 -16 -11/8  —33/8 45/4 1 
31/5 29/5 7/5 1/5 19/5 


Iteration 7: 


1. min{31/5, 29/5, 7/5, 1/5} = 0. 
2. The optimal basic solution and optimal value: 


= (7/5, 0, 4/5, 0, 0, 8/5, 0, 33/5)", f* =—19/5. 


Based on equivalence between the reduced tableau (17.3) and the revised 
tableau (17.19), a revised version of Algorithm 17.3.3 can be stated, matching 
Algorithm 17.3.2. 


Algona 17.3.4 (D- Reduce Phase-I) Initial: B,R,N defined by (17.2). 
Be ; ZN = cn — NBS Top > 0. This algorithm finds a feasible D-reduced 
basis. 


. Compute dy = Ne» — NE(BR’ Bem): 

. Determine column index q € argmaxjey @;. 

. Stop if @ < 0 (infeasible problem). 

. Update zy = Zy + Bay, where B = —Z,/@q. 

. Compute a(R) = Bz ay (R). 

. Determine p € argmaxjer dig (p = is). 

. Stop if dp, q < 0 (feasible D-reduced basis Br obtained). 
. Update Bp! by (17.20). 

. Update (B, N) by exchanging j, and q. 

. Go to step 1. 
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17.4 Bisection D-Reduced Simplex Algorithm 


In Sect. 16.4, the bisection reduced simplex algorithm is described, where the so- 
called existence interval [A, 4], containing the optimal value, is contracted by at 
least a half of the length in each iteration, until an optimal solution is achieved. The 
same idea was also realized from the dual side (Shen, 2006). In this section, we 
show that a more relevant dual bisection algorithm can be derived in the D-reduced 
simplex framework. 

Let [A, wz] be an existence interval. If a dual feasible solution is found with 
objective value f = (A + w)/2, the value is switched up to one associated with 
a dual basic feasible solution; if the latter does not satisfy the optimality condition, 
the objective value is a new lower bound on the optimal value, and hence it is set to 
A. On the other hand, if there exists no dual feasible solution with objective value 
f = (A+ p)/2, the value must be primal feasible. Then, it is switched down to 
one associated with a dual basic solution; if the latter does not satisfy the optimality 
condition, the associated value is a new upper bound on the optimal value, and it 
is then set to 4. Consequently, the resulting [A, jz] is again an existence interval, 
contracted by a half of its length, at least. 

To create an initial existence interval, it uses the D-reduced Phase-I Algo- 
rithm 17.3.3 to produce an upper bound py of the optimal value, and the dual 
D-reduced Phase-I Algorithm 17.2.1 to provide a lower bound i. Then, it bisections 
the interval, until reaching optimality. 

The overall steps are put into the following algorithm. 


Algorithm 17.4.1 (Bisection D-Reduced Simplex Algorithm: Tableau Form) 
Initial: D-reduced tableau of form (17.3). This algorithm solves the D-reduced 
problem (17.1). 


Step 1. Call Algorithm 17.3.3 to seek the primal feasibility: 
(1) stop if returning from step 2 (infeasible problem); 
(2) Returning from step 5: stop if zy > O (the optimality achieved); else set 
ye = f and return. 
Step 2. Call Algorithm 17.2.1 to seek the dual feasibility: 
(1) stop if returning from step 6 (infeasible or lower unbounded); 
(2) if returning from step 2, determine # and column index g such that 


B = —Z,/@q = max{—Z;/@; | @; >0, j € N}. 


Step 3. if 6 4 0, add £ times of the datum row to the bottom row. 
Step 4. stop if @g > 0; dig < 0,i € R (optimality achieved). 
Step 5. setA = f. 

Step 6. set B = (A — y)/2. 

Step 7. Add 6 times of the datum row to the bottom row. 

Step 8. Determine column index q € arg minjey Z;. 

Step 9. If Z, => 0, then: 
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(1) determine 6 and column index gq such that B = —Z/@g, = 
max{—z;/@; |@j > 0, j € N}; 
(2) if 6 ~ 0, add B times of row r to the bottom row; 
(3) stop if ajg < 0, Vie R (optimality achieved); 
(4) goto step 7 witha = f and B = (A — p)/2; 
else 
(5) determine row index p € arg maxjeR Gj gq; 
(6) if apg > 0, convert a), to 1, eliminate the other nonzeros in the column by 
elementary transformations, and go to Step 8; 
(7) add B = —Z,/@zq times of row r to the bottom row; 
(8) stop if zy > 0 (optimality achieved); 
(9) go to Step 7 with setting w = f and B = (u —A)/2. 
end if 


The proof regarding the meanings of the exits of the preceding Algorithm is 
omitted (interested readers is referred to Yuan Shen and P.-Q. Pan 2006) 
We demonstrate the algorithm by the following example. 


Example 17.4.1 Solve following problem by Algorithm 17.4.1: 


min f = 10/7 — 5x, + 7x2 — 6x3 + 5x4, 


st. 2x, + 5x2 —3x34+5x4+ x5 = 1, 
—4x1 — 3x2 + 8x3 — 2x4 + x6 =-l, 

xX} —x2 +2x3+ 6x4 + x7 = 6, 

—7x, + 3x2 + 4x3 — 2x4 +xg = -2, 


xj>0, j=Hl,...,8. 


Answer Initial tableau 


a 1 =f 

{ = «L. 6 1 6 
aj  % 4.13 1] -2* 
=o =e S 10/7 


Iteration 1: 
Arbitrarily take the 4th row as the datum row. Multiply the roe by —1/2, and 
add —1, 1, —6 times of it to rows 1, 2, and 3, respectively, resulting in D-reduced 
tableau: 

Step 1 Call Algorithm 17.3.3 to pursue primal feasibility. r = 4. 
Iteration 2: 
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1. max{7/2, —3/2, —2, 1, -1/2}=7/2 >0, q=1. 
3. Add 6B = —(—5)/(7/2) = 10/7 times of row 4 to the bottom row. 
4. max{—3/2, —1/2, —20} = —1/2 < 0. 
5. Primal feasibility achieved: 
x1 x2 x3 X4 X55 X64 X7 xg RHS 
—3/2 13/2 -1 4 1 1/2 
-1/2 -9/2 6 -1 1 -1/2 
—20 8 13 1 3 
7/2 -3/2 -2 1 -1/2 1 
—5 7 —6 5 —10/7 
xX] x2 x3 X4 X5 X6 X7 xg | RHS 
—3/2 13/2 -1 4 1 1/2 
1/2 -9/2 6 —-l 1 -1/2 
—20 8 13* 1 3 
7/2 3/2 -2 1 -1/2| 1 
34/7 62/7 45/7 5/7 


Step 1(2) w=0. 
Step 2. Call Algorithm 17.2.1 to seek the dual feasibility: r = 4. 
Iteration 3: 
1. min{0, 34/7, —62/7, 45/7, —5/7} = —62/7 < 0, gq =3. 
3. Add 6 = —(—62/7)/(—2) times of row 4 to the bottom row. 
4. max{—1, 6, 13} = 13 > 0, p=3. 
7. Multiply row 3 by 1/13, and add 1, —6, 2 times of row 3 to rows 1, 2, 4, 
respectively: 


Xx] x2 x3 x4 X5 X6 x7 xg RHS 
—79/26 185/26 4 1 1/13 19/26 
227/26* —213/26 -1 1 —6/13 —49/26 
—20/13 8/13 1 1/13 3/13 

11/26 —7/26 1 2/13 —1/26 i} 
—31/2 23/2 2 3/2 —31/7 


Iteration 4: 

1. min{f—31/2, 23/2, 2,0, 3/2} = —31/2 <0, g=1. 

3. Add 6B = —(—31/2)/(11/26) times of row 4 to the bottom row. 

4. max{—79/26, 227/26, —20/13} = 227/26 > 0, p =2. 

7. Multiply row 2 by 26/227, and add 79/26, 20/13, —11/26 times of row 
2 to rows 1,3,4, respectively: 


17.4 


xX x2 
968/227 
1 9 —213/227 
—188/227 
29/227 


18/11 


Iteration 5: 


x3 


x4 
829/227 
—26/227 
—40/227 
238/227 
425/11 


x5 
1 
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X6 
79/227 
26/227 
40/227 

A R/007 


X7 
19/227 
12/227 

S1fe7 
40/227 
62/11 


1. min{18/11, 425/11, 0, 62/11, 1/11} > 0. 


2. return. 
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xg RHS 
17/227 
—49/227 
—23/227 
12/227 1 
1/11 | 2480/77 


Step 2(2) 6B = max{—(18/11)/(29/227), —(425/11)/(238/227), —(62/11)/ 
(40/227), —(1/11)/(12/227)} = —(C./11)/(12/227), ¢q = 8. 
Step 3. Add 6 = —(1/11)/(12/227) times of row 4 to the bottom row: 


xX x2 
968/227 
1 —213/227 
—188/227 
29/227 


17/12 


x3 


x4 
829/227 
—26/227 
—40/227 
238/227 
221/6 


x5 
1 


X6 
79/227 
26/227 
40/227 

== 7927 

1/12 


X7 
19/227 
—12/227 

1227 
40/227 
16/3 


xg RHS 
17/227 
—49/227 
—23/227 
12/227 1 
2561/84 


Step4 a4.g = 12/227 > 0, a1,.g = 17/227 > 0. Step5 4 = —2561/84. 
Step6 Set 6 = (—2561/84 — 0)/2 = —2561/168. 
Step 7 Add 6 times of row 4 to the bottom row: 


x1 x2 
968/227 
i 2213027 
—188/227 
29/227 


—491/925 


Iteration 6: 
Step 8 


x3 


Step9 Z, #0. 


(5) 
(6) 


x4 
829/227 
—26/227 
—40/227 
238/227 
1814/87 


x5 
1 


X6 
79/227 
26/227 
40/227 

=117297 
3164/3849 


1777 times of row | to rows 2,3,4, respectively: 


Iteration 7: 
Step 8 


X7 
—19/227 
—12/227 

S207 
40/227 
1583/598 


X8 


17/227* 


49/227 
—23/227 
12/227 


RHS 


—1432/1777) 2561/168 


min{—491/925, 1814/87, 3164/3849, 1583/598, —1432/1777} 
~1432/1777 < 0, q =8. 


max{17/227, —49/227, —23/227} = 17/227 > 0, p=1. 
Multiply row 1 by 227/17, and add 49/227, 23/227, —12/227, 1432/ 


min{5488/121, 5293/88, 2561/238, 717/157, 1247/714} = 0. 
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Bal x2 x3 X4 x5 XG x7 RHS 
968/17 829/17 227/17 T9/17 = —19/17 
1 193/17 177/17 49/17 19/17 —5/17 
84/17 1 81/17 23/17 11/17 —2/17 
—49/17 —26/17 —12/17  —S5/17 4/17 1 
5488/121 5293/88  2561/238 717/157 1247/714 2561/168 
Step9 zy>0. 


(1) 6B =max{—(1247/714)/(4/17)}, ¢q = 8. 
(2) Add —(1247/714)/(4/17) times of row 4 to the bottom row: 


xX] x2 X3 x4 X5 X6 X7 RHS 
968/17 829/17 227/17 79/17 19/17 
1 193/17 17/17 49/17, 19/17. 5/17 
84/17 1 SIT 23 f1T. Te SBF 
—49/17 SOolt .12F 507, 4/17 1 
267/4 143/2 16 27/4 219/28 


(3) —19/17,-5/17,-—2/17 < 0. Setting xg = 17/4, we obtain the 
optimal basic solution and optimal value: 


x* = (5/4, 0, 1/2, 0, 0,0,17/4,19/4,1)',  f* =—219/28. 


Reference 
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Chapter 18 ®) 
Generalized Reduced Simplex Method peels 


Although we always consider the standard LP problem, the LP problems from 
practice are various. The latter can be transformed into a more general form, that is, 
the so-called bounded-variable LP problem. This type of problem can be obtained 
from the standard problem by replacing the sign restriction with the general bound 
restriction. 

Similarly modifying the reduced problem (16.1), we obtain the reduced bounded- 
variable problem below: 


min f, 


s.t. a(*)=2 l<x<u, eel 


where A c R&"tDX@+D 5 ER"! rank A=m+1, m <n; both/ and u are 
given vectors (may include infinite components). Note that the objective variable 
f is viewed as the same as the (n + 1)th variable x,+1, corresponding to the end 


column an+1 = —em+1- 

To cope with the problem (18.1), in this chapter, we generalize the reduced and 
D-reduced simplex algorithms, deriving the generalized reduced simplex algorithm 
together with the single-artificial-variable Phase-I, generalized dual reduced simplex 
algorithm with the dual Phase-I, and generalized dual D-reduced simplex algorithm. 

Assume that the current basis and nonbasis are as follows: 


B={ji,..., jm}, N=A\B, (n+ DB. (18.2) 
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18.1 Generalized Reduced Simplex Algorithm 


To begin with, we bring up some useful facts under the generalized reduced simplex 
framework. 

It is noticeable that the constraints of (18.1) are equivalent to the following 
canonical form: 


lp <xp=b—Nxy — fanz, Sup, ly < xy <un, 
where 
b=B'b, N=B"'N, Ging) = Bolanyy 
If (x, f) satisfies 


xj=lj or uj, JEN, (18.3) 
lp < ip =b— Nin —Gnyif Sus, (18.4) 
it is called feasible basic solution; if, further, 1p, < xg < ug, it is said 
nondegenerate. If the bounded-variable problem has a feasible solution, it has a 
basic feasible solution, and if it has an optimal solution, then it has an optimal basic 


solution. 
Consider the following line search scheme: 


tp=iptoan, f=f—a, fy =n, (18.5) 


where the stepsize a is determined by the rule below. 


Rule 18.1.1 (Row Rule) Determine the stepsize a and select the row index p such 
that 


a=) =min{a; |i=1,...,m+ ]}, (18.6) 
where 
(uj, — Xj,)/Gi,nt1, fGi,n+1 > 9, 
ai = (Lj, — Xj,)/Gi,n+1> faint <9, i=1,....m+1. (18.7) 


00, if ding =0, 


Lemma 18.1.1 If (, f) is a basic feasible solution, then the solution (X, pf), 
defined by (18.5), is a basic feasible solution, associated with the objective value 
not increasing, or strictly decreasing if x is nondegenerate. 
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Proof Note that xy = xy. For any real number a, the new solution given by 
(18.5) satisfies the equality constraints of (18.1); from (x, f) satisfying the equality 
constraint, in fact, it follows that 
A(T, f)" = Big + Nin + fans 
= Bl&p + oGn41) + Nin + (f ~— @)an41 
= Bxgp+Nxn + Fan 
= b. 


On the other hand, the a determined by (18.6) is clearly nonnegative, so that the 
new solution x satisfies bound constraints ] < x < u. Therefore (x, f) is a basic 
feasible solution. When x is nondegenerate, in addition, it is clear that a > 0, and 
hence 


f _ f 7a< f ’ 
which implies that the objective value decreases. Oo 


It is noted that the decrement of the objective value is equal to the stepsize. The 
a, given by (18.7), is the largest possible stepsize to maintain the feasibility of x. In 
practice, the problem should be deemed as unbounded if a is too large. Note that the 
jp-indexed component of the new solution is equal to the associated upper or lower 
bound, i.e., 


x. [tisiftp aa =O, ms 
=; if Gp.n4i <0. ee 


Jp? 
Introducing the following index sets: 
P={jeNn |x; =1j}, 
M={jeNn|xj =uj}, (18.9) 
on = N'B Tey, 
we define the set 
J={f7 €T | 4p angio; > O}ULZ ET] | Gp npio; < O}. (18.10) 


Then the following is valid for the new solution. 


A 


Lemma 18.1.2 [f the set J is empty, the new solution (x, f), defined by (18.5), is a 
basic optimal solution. 
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Proof By (18.6) and (18.7), it is known that ap, n41 4 0. From the pth equation of 
the canonical form xg = b — Nxy — Gy+1f, it follows that 


f = (bp — oNxN — Xj,)/Ap,n41- (18.11) 
Assuming that (x’, x/,, ,) is any feasible solution, it holds that 


Ij <x; <uj, j=l,...,n. (18.12) 
If Gp,n+i1 > 0, then J = G implies that 


or <0, of = 0. (18.13) 


and from the second expression of (18.13) and <n < uy = xm, it follows that 
onxy < opin. 
Combining the preceding two expressions and dp, +1 > 0 gives 
— (yxy) /Gp,n41 = —(GNEN)/Gp,n41 = —(ONdn)/Gp,n41- (18.14) 
On the other hand, from (18.8), (18.12), and @p,n41 > 0, it follows that 
- x,/Gp,n+1 > —Uj,/4p,n+1 = —Xj,/4p,n41- (18.15) 


Therefore, by (x, f) and (x’, I i) satisfying (18.11), (18.14), and (18.15), it holds 
that 


/ hb Ty / = r Ta im = a 
Xny1 = (bp — OnXy — X},)/4p,nt+1 = (bp — OnXN — Xj,)/Gp,n+1 a ee 


Therefore, by considering Lemma 18.1.1, it is known that % is an optimal basic 
solution. 
It can be shown analogously that the statement is valid if dp, n41 < 0. oO 


In the case of J # G, the following rule is well-defined. 
Rule 18.1.2 (Column Rule) Select the column index 


q € argmax |o;|. (18.16) 
Jed 
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Then after the associated basis change, when j, left from and g entered to the 
basis, an iteration is complete. It is clear that the new basis corresponds to the new 
solution (x, x), given by (18.5) together with (18.6) and (18.7). 

The overall steps can be put into the following algorithm. 


Algorithm 18.1.1 (Generalized Reduced Simplex Algorithm) Initial: (B, N), 
Bo!; Gag) = —Bo'em41; basic feasible solution x. This algorithm solves the 
reduced bounded-variable problem (18.1). 


I. Determine the stepsize a and row index p by (18.6) and (18.7). 
2. Ifa #0, update xp = XB + AAn41. 

3. Compute on = NTB Tey, 

4. Stop with f = c'x if J defined by (18.10) is empty. 

5. Determine column index q by (18.16). 

6. Compute ag = Bo'ag and v = —Gp,n41/Oq- 

7. Ifv £0, update Gn41 = Gn41 + Vag — ep). 

&. Update a. 

9 


. Update (B, N) by exchanging jp and q. 
10. Go to step 1. 


Theorem 18.1.3 Algorithm 18.1.1 produces a sequence of basic feasible solutions. 
Assuming degeneracy, it terminates at step 4, offering a basic optimal solution. 


Proof The validity is from Lemmas 18.1.1 and 18.1.2 and the discussions preceding 
Algorithm 18.1.1. oO 


Example 18.1.1 Solve the following problem by Algorithm 18.1.1: 


min f = —x, — 3x2 — 2x3 + 4x4 — 5x5, 
s.t. —2x, +x2 —3x3 -—x4 +2x5 +2X6 = —52, 
3x, —3x2 +x3 +5x4 —2Xx5 +x7 = 26, 
x, —6x2 —2x3 —4x4 +6x5 t+xg= 7, 
2<x,<10, -5<x.<12, 1<4x3< 15, 
—4<x4< 27, -3<x5< 14, —-9< x6 < 18, 
—7<x7< 13, -l10< xg < 21. 


Given a feasible solution, x = (2, —5, 15, —4, —3, 1, 4, 7). a =—-18. 


Answer The initial tableau is 


x] x2 x3 x4 x5 X6 x7 Xg f f = -18 
—2 1 3 -1 1 1 —52 

3 —3 1 5 —2 1 26 

1 —6 —2 —4 6 1 7 
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Iteration 1: Convert the preceding to a reduced tableau by taking p = 4 and q = 1. 
Multiply row 4 by —1, and then add 2, —3, and —1 times of row 5 to rows 1, 2, and 
3, respectively: 


Xx] x2 x3 X4 x5 X6 X7 x8 f f = -18 
7 1 -9 11 1 2 —52 
—12 —5 17 -17 1 -3 26 
—9 —4 1 1 1 7 
1 3 2 —4 5 1 


Taking the left side of the preceding tableau as original data, call Algo- 
rithm 18.1.1. 
Initial: B = {6,7,8, 1}, N = {2,3,4,5}, BI =TJ, xy = (-S(y, 15-4, 
—3(_y)', xp =(1, 4,7, 2)', f= 18. 29 =(2,-3,-1, 1). The x is a basic feasible 
solution. 
Iteration 2: 


1. w = min{(18 — 1)/2, (—7 — 4)/(—3), (—10 — 7)/(—1), (10 — 2)/(1)} 
= min{17/2, 11/3, 17, 8} = 11/3, p =2. 
. ¥B=(1,4, 7, 2)? +11/3)(2, —3, -1, 1) T= (25/3, —7, 10/3, 17/3)". 
oy = N'B Tey = (—12, —S, 17, -17)". 
J = {2,54 0. 
. max{| — 12],|—17|} =17, gq =5. 
. Gs = B~'as = (11, —17, 1,5)". 
. vb =—(—3)/(-17) = -3/17, 
Gn41 = (2, -3,-1, 1)? + (-3/17)(11, —17— 1,1, 5)? 
= (1/17, 3/17, —20/17, 2/17)". 
1 11/17 
—1/17 
1/17 1 
Sg 1 
9. xg = (25/3, —3, 10/3, 17/3)', B = {6,5, 8, 1}, 
tn = (—5(-), 1S, —4_), -7(5)7, N = {2, 3,4, 7}. 


8. B= 


Iteration 3: 


1. a@=min{(18—(25/3))/(1/17), (14—€3))/(3/17), (-10—(10/3)) /(—20/17), 
(10 — (17/3))/(2/17)} = min{493 /3, 289/3, 34/3, 221/6} = 34/3, p =3. 
2. ¥p = (25/3, —3, 10/3, 17/3)? + (34/3)(1/17, 3/17, —20/17, 2/17)7 
= (9, -1, -10, 7)". 
3. oy = N™B-Te3 = (—165/17, —73/17, 1, 1/17). 
4, J ={2} 49. 
5. max{| — 165/17||| = 165/17, g =2. 
6. G = Boa = (-13/17, 12/17, -165/17, —9/17)". 
7. v = —(—20/17)/(—165/17) = —4/33, 
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Gnu = (1/17, 3/17,-20/17, 2/17)? + (4/33)(-13/17, 12/17, 165/17 — 
1,-9/17)7 
= (5/33, 1/11, 4/33, 2/11)". 


1 —13/165 \ /l 11/17 1 106/165 —13/165 

g gota | 14/5 7 _| -3/55 4/55 
—17/165 RAT A 1/165 —17/165 
—3/55 5/171 16/55 —3/55 | 


9. xg = (9,-1,-5,7)', B = {6,5, 2, 1}, 
kn = (-10), 15, —4(_), -7(_))?, N = {8, 3, 4, 7}. 


Iteration 4: 


1. a = min{(18 — 9)/(5/33), (4 — (—1))/C1/11), (12 — (—5))/(4/33), 
(10 — 7)/(2/11)} 
= min{297/5, 165, 561/4, 33/2} = 33/2, p =4. 
2.x%—8 = (9,-1,—5,7)' + (33/2)(5/33, 1/11, 4/33,2/11)’ = (23/2, 1/2, 
—3, 10). 


3. oy = N'B Teg = (—3/55, 42/55, 52/55, 16/55)". 
4.3 ={4,7} 49. 
5. max{|52/55|, |16/55|} = (52/55, 16/55} = 52/55, g = 4. 
6. a4 = Bo!aq = (317/165, —51/55, —17/165, 52/55)". 
7. v = —(2/11)/(52/55) = —5/26, 
G@na41 = (5/33, 1/11, 4/33, 2/11)" + C5/26)(317/165,-51/55,-17/165, 
52/55—1)" 
= (—17/78, 7/26, 11/78, 5/26)". 
1 —317/156 1 106/165 —13/165 12/39 5/156 —317/156 

8 ple 1 51/52 —3/55 4/55 _| 3/13 1/52 51/52 

1 17/156 —1/165 —17/165 1/39 —17/156 17/156 


55/52 16/55 —3/55 1 4/13. —3/52 55/52 
9. xp = (23/2, 1/2, -3, —4)7, B={6, 5, 2, 4}, 
Xn =(—10(-), 15%, 10%, —7_))7, NW = {8, 3, 1, 7}. 


Iteration 5: 


1. a = min{(—9 — 23/2)/(—17/78), (14 — (1/2))/(7/26), (12 — (—3))/(11/78), 
(27 — (—4))/(5/26)} 
= min{1599/17, 351/7, 1170/11, 806/5} = 351/7, p =2. 
. Xp = (23/2, 1/2, —3, —4)? + (351/7)(—17/78, 7/26, 11/78, 5/26)" 
= (4/7, 14, 57/14, 79/14)". 
. on = N™BTey = (1/52, 19/26, 51/52, 3/13)!. 
_ JF = {8,4 GO. 
. max{|1/52|, |3/13|]} = 3/13 > 0, gq =7. 
. a7 = Boaz = (2/39, 3/13, 1/39, 4/13)7. 
. v = —(7/26)/(3/13) = —7/6, 
Gn41 =  (—17/78,7/26, 11/78, 5/26)’ + (—7/6)(2/39, 3/13 — 
1, 1/39, 4/13)" 


N 


NAN AB WwW 


468 18 Generalized Reduced Simplex Method 


= (—5/18, 7/6, 1/9, —1/6)'. 


1 -2/9 12/39 5/156 —317/156 1 1/36 —9/4 

¢oa| BA 3/13 1/52 51/52 _]| 1 12 17/4 
-1/91 1/39 -17/156 17/156 -1/9 
4/3 1 4/13 —3/52 55/52 apa 


9. xp = (4/7, -—7, 57/14, 79/14)", B = {6,7, 2, 4}, 
xn = (-10-), 15, 107, 14)7, NW = {8, 3, 1, 5}. 


Iteration 6: 


1. a = min{(—9 — 4/7)/(—5/18), (13 — (—7))/(7/6), (12 — (57/14))/(1/9), 
(9 = (9/1 (= 15) 

= min{1206/35, 120/7, 999/14, 363/7} = 120/7, p = 2. 
. Xp = (4/7, -—7, 57/14, 79/14)! + (120/7)(—5/18, 7/6, 1/9, —1/6)" 

= (—88/21, 13, 251/42, 39/14)". 
. oy = N™B Te, = (1/12, 19/6, 17/4, 13/3)". 
_ JF = {8} 40. 
. max{|1/12|} = 1/12 > 0, g =8. 
ag = B-1ag = (1/36, 1/12, —1/9, —1/12)". 
. v =—(7/6)/(1/12) = —14, 

Gn+1 = (—5/18, 7/6, 1/9, —1/6)' + (—14)(1/36, 1/12—1, —1/9, —1/12)" 
= (—2/3, 14, 5/3, 1)". 


N 


1 —4/13 1 1/36 —9/4 1 —4/13 1/468 —185/52 
7 13 1 1/12 17/4 7 2 1 51 
4/3 1 -1/9 4/3 17/3 
1 1 ~1/12 -1/4 1 4 


9. Xp = (—88/21, —10, 251/42, 39/14)", B = {6, 8, 2, 4}, 
ky = (138, 15, 10, 14) TW = {7, 3, 1, 5}. 


Iteration 7: 


1. a = min{(—9 — (—88/21))/(—2/3), (21 — (—10))/14, (12 — (251/42))/(5/3), 
(27 — (39/14))/1} 
= min{101/14, 31/14, 253/70, 339/14} = 31/14, p =2. 
2. Xp = (—88/21, —10, 251/42, 39/14)? + (31/14)(—2/3, 14, 5/3, 1)? 
= (—17/3, 21, 29/3, 5)'. 
3. oy = N™B-Te, = (1, 166, 51, 52)7. 
4. J =. The optimal basic solution and optimal value are 


x* = (10, 29/3, 15, 5, 14, -17/3, 13, 21)", 
f* =(-1, -3, —2, 4, —5)(10, 29/3, 15, 5, 14)’ = —119. 


Finally, we formulate the tableau version of Algorithm 18.1.1 based on the 
relationship between the reduced and the revised reduce simplex tableaus. 
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Algorithm 18.1.2 (Generalized Reduced Simplex Algorithm: Tableau Form) 
Initial: reduced tableau of form (18.19), associated with (x, f) with (18.3) and 
(18.4) satisfied. This algorithm solves the reduced bounded-variable problem 
(18.1). 


1. Determine stepsize a and row index p by (18.6) and (18.7). 

2. Ifa #0, update Xp = Xp + aan41 and f = f —a. 

3. If J ={j €T | ap nzidpj > O}ULF € TL | Gp nt1apj < O} = Y, compute 
f = c'x, and stop (optimality achieved). 

4, Determine column index q € argmaxjey |Gp j\- 

5. Convert pq to 1, and eliminate the other nonzeros in the column by elementary 
transformations (see the Note below). 

6. Go to step 1. 


It is noted that the preceding step 5 does not touch the xg column but replaces its 
pth component by xq. 


18.1.1 Generalized Reduced Phase-I: Single Artificial Variable 


In this section, we derive a generalized reduced Phase-I procedure similar to 
the single-artificial-variable Algorithm 14.2.1 for the standard problem. Such an 
algorithm is especially suitable in the reduced simplex framework. 

Assume that x satisfies 


xj =1j oruj, JEN, (18.17) 


but not 


For given xg such that /g < xg < ug, set the artificial f’ (= xn+41) column as 
follows: 


Gnt1 =b—Nxy—Xp, f' =1. (18.18) 
Construct the following auxiliary reduced tableau: 


1 1 ¢i 

7 ae BE ee (18.19) 
I N an+i XB 

This tableau is clearly feasible, from which the generalized reduced simplex 

algorithm can get itself started. If the optimal objective value vanishes, then a 

feasible reduced tableau of the original problem is obtained, and hence it is ready to 
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go to Phase-II. To pursue zero optimal value, we modify Algorithm 18.1.2, giving 
the following algorithm. 


Algorithm 18.1.3 (Tableau Generalized Reduced Phase-I: Single Artificial 
Variable) Initial: reduced tableau in form (18.19). X,Gn41, f’ = 1 satisfying 
(18.17) and (18.18). This algorithm finds a feasible reduced tableau for the 


bounded-variable problem (18.1). 


i. Determine stepsize a and row index p by (18.6) and (18.7). 


2. Ifa > f', seta= fi’. aad 
3. Ifa #0, update Xp = XB + @an+) and f' = f'— a. 
4. Stop if f' =0 (feasibility achieved). 
5. Stop if 
J={7 ET | Gp np1apj > O}FULZ ET | Gp npidp;j < OJ =D 
(infeasible problem). 


6. Determine column index q € arg maxjey |dpj\. 
7. Convert ap q to 1, and eliminate the other nonzeros in the column by elementary 
transformations 
(without touching x column, only its pth component is replaced by Xq). 
8. Go to step 1. 


Example 18.1.2 Solve the following problem by Algorithm 18.1.2, supported by 
Algorithm 18.1.3 in Phase-I: 


min f = —7x; + 6x2 + 3x3 — 4x4, 


s.t. —2x, —3x2 +7x3 —5x4 +x5 = 2». 
3x, —5x2 —2x3 —Xx4 +X6 =-15, 
—2x, —4x2 +3x3 —6x4 +Xx7 = QO, 
—5x, +6x2 —3x3 +3x4 t+xg= 7, 
—7 <x, <6, -—-3<212 <9, -5< x3 < 12, 
—4<x4< 11, -—7<x5<5, -8< x6 < 14, 
—7<x7<7, -6< xg < 10. 


Answer B = {5,6,7,8}, N = {1,2,3,4}. Set xg = (0,0,0, 0)? and xy = 
(-7-), 9, 12), —4(_))". Form the auxiliary objective column 


ag = (2, -15, 0,7)! — (91, —86, 38, 41)? = (—89, 71, —38, —34)!, 


and set the initial auxiliary objective value f’ = 1. 
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The initial auxiliary reduced tableau is as follows: 


= 


™~ 


Phase-I: call Algorithm 18.1.3. 
Iteration |: 


1. a = min{(—7 — 0)/(—89), (14 — 0)/(71), (—7 — 0) /(—38), (—6 — 0)/(—34)}; 
= min{7/89, 14/71, 7/38, 3/17} = 7/89, p= 1. 


. 7/89 < fl =. - 
. ¥B = (12, 497/89, —266/89, —238/89, 0)", f’ = 1 — 7/89 = 82/89. 
_J={1,3,44 490. 


. max{| — 2, |7|,|-—5|}=7, ¢g =3. 

. Multiply row | by 1/7, and then add 2, —3, 3, and —3 times of row 1 to rows 2- 
5, respectively (without touching xg column, only its first component is replaced 
by x3 = 12): 


AUD NW Pb 


f' = 82/89 
—3/7 —5/7 1/7 12 
41/7 -17/7 2/7 497/89 
—19/7 —21/7 —3/7 —266/89 
33/7 6/7 3/7 —238/89 


51/7 —13/7 —3/7 


—7 497/89 —266/89 —238/89 82/89 
8 7 6 0 


Iteration 2: 


1. w = min{(—5 — 12)/(—89/7), (14 — 497/89) /(319/7), (7 — (—266/89))/(1/7), 
(—6 — (—238/89))/(—505/7)} 
= min{119/89, 306/1657, 6223/89, 107/2321} = 107/2321, p = 4. 
2. 107/2321 < f’ = 82/89. 
3. Xp = (5764/505, 5764/505, —1506/505, —7, 0), f’ = 82/89 — 107/2321 = 
442/505. 
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5. J= {1,2} 49V. 

6. max{| — 41/7], |33/7|} = 41/7, q=1. 

7. Multiply row | by —7/42, and then add 2/7, —17/7, 8/7, and 43/7 times of row 
1 to rows 2-5, respectively (without touching xg column, only its 4th component 
is replaced by x} = —7): 


1 —31/41 5/41 377/41 5764/505 
—160/41* 85/41 19/41 17/41 642/41 5764/505 
149/41 165/41 —21/41 8/41 583/41 —1506/505 
1 33/41 6/41 3/41 7/41 505/41 -7 

96/41 113/41 —36/41 —43/41 

4 7 10 

3881 __ 1506 —6 

305 305 


8 7 6 


= 


~ 


Iteration 3: 


1. a = min{(—5 — 5764/505)/(—377/41), 4 — 3881/505) /(642/41), 
(7 — (—1506/505))/(583/41), (6 — (—7))/(505/41)} 
= min{1171/656, 221/548, 245/349, 533/505} = 221/548, p = 2. 

2. 221/548 < f’ = 442/505. 

3. XB = (1649/214, 9, 3834/1393, —3169/1559, 0)", f’ = 442/505 — 221/548 = 
101/214 40. 

5. J = {2,5, 8} #@. 

6. max{| — 160/41], [19/41], |17/41|} = 160/41, g =2. 

7. Multiply row 2 by —41/160, and then add 27/41, 149/41, 33/41, and —96/41 
times of row 2 to rows 1, 3, 4, and 5, respectively (without touching xg column, 
only its second component is replaced by x2 = 9): 


f’ = 101/214 

13/32 7/160  —27/160 19/160 —947/80 | 1649/214 

17/32 —19/160 41/160 —17/160 —321/80 9 

—67/32 —151/160 —149/160 1 —93/160 —29/80 | 3834/1393 

9/32 —27/160 —33/160 —41/160 727/80 | —3169/1559 
—4/5 


~ es 
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Iteration 4: 


1. a = min{(—5 — 1649/214)/(—947/80), (—3 — 9)/(—321/80), 
(—7 — 3834/1393) /(—29/80), (6 — (—3169/1559))/(727/80)} 
= min{3498/3259, 320/107, 3605/134, 5803/6565} = 5803/6565, p = 
2s 
2. a> f’ = 101/214; set a = 101/214. 
3. Xg = [339/160, 1137/160, 4845/1877, 4605/2041]", f’ = f’ a =0. 
4. The feasibility is achieved. 


Phase-II: 

Replacing the artificial f’ column by the f column of the original problem, 
and setting f = c'x = (—7, 6, 3, —4, 0, 0, 0, 0)(4605/2041, 1137/160, 339/160, 
—4, —7, 14, 4845/1877, —6)! = 246/5, we obtain the following feasible tableau: 


SONS 


—13/32 7/160 27/160 —19/160 339/160 
17/32 -19/160 —41/160 —17/160 1137/160 
—67/32 —151/160 149/160 —93/160 4845/1877 
9/32 -27/160 —33/160 —41/160 4605/2041 

—3/5 —4/5 - 


6 9 12 


4605 1137-339 
2041 160 160 


7 3 5 


~ eS 


Take p = 5. J = {4,5, 6, 8} 4 O. max{| —4], | —3/5], [3/5], |-—4/5|}=4, g = 
4. 

Multiply row 5 by —1/4, and then add 13/32, —17/32, 67/32, and —9/32 times 
of row 5 to rows 1, 2, 3, and 4, respectively, leading to the following feasible reduced 
tableau: 


Fa 26/5 
67/640  —147/640 —3/80 13/128 | 339/160 
—127/640 —113/640 —17/80 —17/128 | 1137/160 
—403/640 —797/640* 1 13/80 67/128 | 4845/1877 
—27/128 —21/128 —5/16  —9/128 | 4605/2041 
3/20 —3/20 4 


9 12 


= 


1137 339 
T60 = T60 
3 5 


™~ 
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Call Algorithm 18.1.2. 
Iteration 5: 


1. a = min{(12 — 339/160) /(13/128), (—3 — 1137/160)/(—17/128), 
(7 — 4845/1877) /(67/128), (—7 — 4605/2041) /(—9/128), 
(11 — (—4))/(1/4)} 
= min{6324/65, 6468/85, 3191 /378, 5924/45, 60} = 3191/378, p = 
3. 
3. xp = (997/335, 401/67, 14, 1252/753, —2857/1512)", 7 = 246/5 — 
3191/378 = 3709/91. 
4. J={6} 49%. 
5. max{| — 797/640|} = 797/640, ¢ = 6. 
6. Multiply row 3 by —640/797, and then add 147/640, 113/640, 21/128, and 
3/20 times of row 3 to rows 1, 2, 4, and 5, respectively 


(without touching xg column, only its 3rd component is replaced by x6 = 14): 


XI x2 x3 x4 X5 X6 X7 Xg f | f= 3709/91 
1 176/797 —147/797 = —6/797 4/797 | 997/335 
1 —87/797 —113/797 —151/797* —165/797 | 401/67 
403/797 1  -—640/797 104/797 —335/797 14 
1 —102/797 =105/797 9 —232/797 —111/797 | 1252/753 
1 180/797 —96/797 175/797 149/797 | —2857/1512 
u 6 9 12 11 5 14 7 10 
a ee a -6 
I 7 3 5 4 7 8 7 =6 


Iteration 6: 


1. a = min{(12 — 997/335)/(4/797), (—3 — 401/67) /(—165/797), 
(—8 — 14)/(—335/7797), (—7 — 1252/753)/(—111/797), 
(11 — (—2857/1512))/(149/97)} 
= min{219357/7122, 13758/317, 7537/144, 33028/531, 11514/167} 
= 13758/317, p =2. 


3. tp = (527/165, 6, —140/33, —241/55, 1027/165)', f = 3709/91 — 
13758/317 = —3628/1373. 
4, J = {5,8} 40. 


5. max{| — 87/797], | — 151/797|} = 151/797, q =8. 
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6. Multiply row 2 by —797/151, and then add 6/797, —104/797, 232/797, and 
—175/797 times of row | to rows 1, 3, 4, and 5, respectively (without touching 
xg column, only its second component is replaced by xg = —6): 


f = —3628/1373 
-6/151 1 34/151 27/151 2/151 527/165 
—797/151 87/151 113/151 1 165/151 -6 
104/151 65/151 1 136/151 85/151 —140/33 
—232/151 6/151 13/151 27/151 241/55 
175/151 1 15/151 —43/151 8/151 1027/165 


12 11 5 
527 1027, _7 
Tes TO 


= 


~ 


5 4 de 


Iteration 7: 


1. w@ = min{(12 — 527/165)/(2/151), (10 — (—6))/(165/151), 
(—8 — (—140/33)) /(—85/151), (6 — (—241/55))/(27/151), 
(—4 — 1027/165)/(—8/151)} 
= min{4654/7, 2416/165, 5981/896, 2845/49, 11000/57} 
= 5981/8396, p = 3. 
3. ¥g = (279/85, 22/17, —8, 271/85, 499/85)", f = —3628/1373 — 
5981 /896 = —2963/318. 


f = —2963/318 
—6/151 1 34/151 27/151 2/151 279/85 
~797/151 87/151 113/151 1 165/151 22/17 
104/151 65/151 1 136/151 —85/151 -8 
1 —232/151 6/151 13/151 27/151 271/85 
175/151 15/151 —43/151 —8/151 499/85 


~ as 


4. J = @. The optimal basic solution and optimal value are 
x* = (—271/85, —3, 279/85, 499/85, —7, —8, 7, 22/17)", 


f* = —2963/318. 
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18.2 Generalized Dual Reduced Simplex Algorithm 


The generalized reduced Algorithm 18.1.2 pursues the dual feasibility while 
maintaining primal feasibility. Assume that reduced simplex tableau is dual feasible, 
i.e., for some row index p, it holds that 


J={f ET | ap ntiapj > O}ULF ET | Gp, n41ap j < OJ =9 (18.20) 


and Xj, = lip if dp,nti < 0, while Xj, = Uj, if dp nti > 0. 
Introduce the following bound violations: 


Vj, — Xj, thx < Ti, 
Pi = 4 Uj, —X;j,, WX; > uj, i=1,...,.m+1. (18.21) 
0, if lj, < Xj S Uj, 


Then, seeking the primal feasibility while maintaining dual feasibility, we derive 
the following dual algorithm. 


Algorithm 18.2.1 (Generalized Dual Reduced Simplex Algorithm: Tableau 
Form) Initial: dual feasible reduced tableau of form (18.19). This algorithm 
solves the bounded-variable problem (18.1). 


I. Select row indexr € argmax{|p;| |i = 1,...,m +1, i # p}, where pj; is 
defined by (18.21). 

2. Stop if p, = 0 (optimality achieved). 

3. Set p =r, and go to step 7 if 


J ={j €T | sign(pr)ayj < O}U Lj € T1| sign(o,)a, j > 0) =H. 


4. Determine index q such that B = |dpq/@rq| = minjey |4p j/G, j\- 

5. Convert a; q to 1, and eliminate the other nonzeros of the column by elementary 
transformations (without touching the xp column, except for its rth component 
replaced by the value of Xq). 

. Update xg = Xp — prj,» Xj, — Xj, + Pr. 

Compute a = —pp/Ap,n+1- 

. Stop if a < 0 (dual unbounded or infeasible problem). 

If a #0, set Xp — n+, and f = f +a. 

. Go to step 1. 


SCaND 


SN 
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Practically, the bound flipping tactic should be incorporated into Algo- 
rithm 18.2.1, with its revised version formed therewith. 


Example 18.2.1 Solve the following problem by Algorithm 18.2.1: 


min f = xj — 2x2 +5x3+ x4 


s.t. 3x7 —2x2 +23 —- 4444+ x5 = —3, 
6x, + 3x2 — 3x3 + 6x4 + x6 = 4, 
—5x, — 4x2 — 6x3 + 2x4 + x7 =--7 
O<x,<10, IL<x2.<10, -—20< 4x3 <5, 


—4<xg<7, O< x5, x6,x7 < 15. 


Answer B = {5,6,7}, N = {1, 2,3, 4}. Set xy = (Oy, 10), —20(-), —4.)7, 
then 


3 3-2 1-4 : 21 
Xp=b—Nxyn = 4]—- 6 3-3 6 20 =] —-62 |], 
—7 —5-4-6 2 —79 
—4 
f = (1, —2, 5, 1)(0, 10, —20, —4)™ = —124. 
Initial tableau: 
XxX] x2 X3 X4 x5 X6 XT f f = 124 
3 —2 1 —4 1 21 
6 3 —3 6 1 —62 
—5 —4 —6 2 1 —79 
} 1* —2 5 1 —1 124 
u | 10 10 5 7 15 15 15 +00 
x 0 10 —20 —4 21 62 79 124 
I 0 1 —20 —4 0 0 0 —0oo 


Iteration |: Turn the preceding to the reduced tableau by taking a4,; at the pivot. 
Add —3, —6, and 5 times of row 4 to rows 1, 2, and 3, respectively (without touching 
the xg column, but its 4th component is replaced by the value, 0, of x1). 
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™~ 


which is a dual feasible reduced tableau with r = 4 and p = 4. 
Iteration 2: 


1. 


3. 
7. 


™~ 


op, = 15-21 =—-6, p2 = 0 — (—62) = 62, p3 = 0 — (—79) = 79. 
max{| — 6], |62|, |79|} = 79, r =3. 

J=6, p=3. 

a = —79/(—5) > 0. 


.xp = (21,—-62,—79,0)' — (79/5)(3,6,—5,—-1)' = (—132/5, -—784/5, 


0, 79/5)" 
f = -124479/5 = —541/5. 


—784/5 0 —541/5 
0 0 —oo 


Iteration 3: 


1. 


RW 


p1 = O—(—132/5) = 132/5, p» = O—(—784/5) = 784/5, pa = 10—79/5 = 
—29/5. 
max{|132/5], |784/5|, | — 29/5]} = 784/5, r =2. 


ed S253}: 
. min{|(—14)/15], |19/(—33)|} = 19/33, ¢ = 3. 
. Multiply row 2 by —1/33, and add 14, —19, —5 times of row 2 to rows 1,3,4, 


respectively (without touching the xg column, but its 2nd component is replaced 
by the value, —20, of x3). 


. ¥g = (—132/5, —20, 0, 79/5)? — (784/5)(—14/33, —1/33, 19/33, 5/33) 


= (1324/33, —2516/165, —14896/165, —1313/165)", x» = 0. 


. & = —(14896/165)/(—17/11) = 5199/89 > 0). 


18.2 Generalized Dual Reduced Simplex Algorithm 479 


gp = (1324/33, —2516/165, —14896/165, —1313/165)" — (5199/89) (5/11, 
9/1, 17/1, 1D! 

= (502/37, —1828/395, 0, -1771/669)', f = —541/5 + 5199/89 = 
—846/17. 


™~ 


Iteration 4: 


1. 


3} 
4. 


P1, p2 = 0, p4 = 0 — (—1771/669) = 1771/669, 
max{| — 1771/669|}, r = 4. 
J = {2}. 
min{|(3/11)/(—59/1D)}, ¢ = 2. 
. Multiply row 4 by 11/3, and add 26/11,5/11,59/11 times of row 4 to rows 
1,2,3, respectively 
(without touching the xg column, but its 4th component is replaced by the 
value, 10, of x2). 
. ¥g = (502/37, —1828/395, 0, 10)" — (1771 /669) (26/3, 5/3, 59/3, 11/3)? 
= (—8822/941, —3851/426, —15046/289, 589/2007)', x; = 0. 
. & = —(38992/749) /(—10/3) = 19319/1237 > 0). 
.¥g = (—8822/941, —3851/426, —15046/289, 589/2007)' — (19319/1237) 
(—1/3, —1/3, —10/3, —1/3)" 
= (—15504/3721, —2166/565, 0, 11/2)", f = —846/17 + 19319/1237 = 
—205/6. 


= 


™~ 
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Iteration 5: 


. pt = 97125/25337, 2, p4 = 0. max{|97125/25337|}, r = 1. 
_J=6, p=1. 
. &@ = —(97125/25337)/(—1/3) = 325662/26053 > 0. 
. ¥B = (—15504/3721, —2166/565, 0, 11/2)? — (325662/26053)(—1/3, —1/3, 
—10/3, —1/3)" 
= (0, 1/3, 125/3, 29/3)", f = —205/6 + 325662/26053 = —65/3. 


ON We 


29/3 1/3 aa 125/3 65/3 
1 = | 0 —oo 


_~ es 


Iteration 6: 


» 2, 02, P4 =O, 03 = 15 — 125/3 = —80/3, r = 3. 
. J{1, 4, 6} 4B. 
- min{|(26/3)/(59/3)|, \(5/3)/(80/3)1, 1(8/9)/(32/9)|} = 5/3)/(80/3), q = 4. 
. Multiply row 3 by 3/80, and add —5/3, —5/3, —11/3 times of row 3 to rows 
1, 2, 4, respectively (without touching the xg column, but its 3rd component is 
replaced by the value, —4, of x4). 
6. xp = (0,1/3, —4,29/3)' — (—80/3)(—1/16, —1/16, 3/80, —11/80)' = 
(—5/3, —4/3, —3, 6)', x7 = 125/3 — 80/3 = 15. 
7. a = —(5/3)/(—1/8) = 40/3 > 0). 
9.%—p = (—5/3,—4/3,—-3,6)' — (40/3)(—1/8, —1/8,—1/8,1/8)' = 
(0, 1/3, —4/3, 13/3)", 
f = —65/3 + 40/3 = —25/3. 


an BR We 


119/16 2/3 —-1/16 —1/8 
7/16 0 -1/16 —I/8 
59/80 2/15 3/80 -1/8 


77/80 1/15 -11/80 ‘1/8 
10 5 7 
13/3 1/3 4/3 0 
1 S90) ied 9 


= 


™~ 
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Iteration 7: 


1. p2, 03, p4 = 0. 
2. Terminated with optimal solution and objective value: 


x* = (0, 13/3, 1/3, —4/3, 0,0, 15)", f* = —25/3. 


18.2.1 Generalized Dual Reduced Phase-I 


When the components of / and wu are all finite, it is, in principle, possible to assign 
appropriate values to nonbasic variables such that the tableau is dual feasible. 
Numerically, however, too large a bound value is unacceptable, let alone the 
magnitude of some components of bounds may be infinite. Therefore, it is still 
required to provide an initial dual feasible tableau or solution for the generalized 
dual reduced algorithm. 

A choice is to directly modify the reduced simplex Algorithm 18.1.2, ignoring 
the requirement of the initial reduced tableau being feasible. The overall steps are 
as follows. 


Algorithm 18.2.2 (Generalized Dual Reduced Phase-I: Tableau Form) Initial: 
reduced tableau of form (18.19) and solution x. This algorithm finds a dual feasible 
tableau to the bounded-variable problem (18.1). 


1. Determine stepsize a and row index p by (18.6) and (18.7). 

2. Ifa £0, set Xp = Xp +adn4, and f = f —a. 

3. Stop if J ={j €V | Gp np14pj > O}ULF € | ap npidpj < OJ =O 
(dual feasibility achieved). 

4, Determine column index q € argmaxjey |Gp j\- 

5. Convert Gpq to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

6. Go to step 1. 


Example 18.2.2 Find a dual feasible solution to the following problem by Algo- 
rithm 18.2.2: 


min f = 4x1 — 5x2 +2x3+ x4, 


S.t. 3x, —4x2 +43 —5x44+ x5 = —3, 

— 2x, — 6x2 — 3x3 + 6x4 + x6 = 26, 

— 6x, + 2x2 — 5x34 3x4 +x, = 21, 

—0o <x 59, -T<x9<to, —-1<%43< 12, 
O<x4<8, O<x5<10, -S5<x6 <6, 


0 < x7 < +o. 
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Answer Set xy = 9, —7), -1, O¢-y)™ (N = {1,2,3,4). ip = b- 
Niy = (—3, 26,21)? — (64,27, -63, 69)? = (—57,-1,84)", f = chip = 


(4, -5, 2, 1)7(9, -7, -1, 0) = 69, B = {5, 6, 7}. 


Initial: reduced tableau (where the RHS column is listed to be easier to go to Phase- 
ID: 


~ IS 


Iteration 1: Turn the preceding to a reduced tableau. 
Take p= 4, J — {1,2} AW. max{|4],| —5]} =5, g =2. 

Multiply row 4 by —1/5, and then add 4, 6, and —2 times of row 4 to rows 1, 2, 
and 3, respectively 

(without touching xg column, only its 4th component is replaced by x2 = 7). 


3/5 29/5 
~27/5 24/5 
2s. 1775 


2/5 1/5 


= 


™~ a 


Call Algorithm 18.2.2. 
Iteration 2: 


1. a = min{(10 — (—57))/(4/5), (6 — (—1))/(6/5), 
(0 — 84)/(—2/5) (co — (—7))/(1/5)} = 35/6, p = 2. 
2. Xp = (—157/3, 9, 245/3, —35/6)', f = 69 — 35/6 = 379/6. 
3. J ={1, 449. 
4. max{| — 34/5], |24/5|]} = 34/5, q =1. 
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5. Multiply row 2 by —5/34, and then add 1/5, 22/5, and 4/5 times of row 2 to 
rows I, 3, and 4, respectively (do not touch xg column, except for the second 
component being replaced by x; = 9): 


15/34 —101/17 1 —1/34 
27/34. 12/17 —5/34 
—12/17* 5/17 Se A 


4/17 -13/17 —2/17 
10 6 
-157/3 6 245/3 
0 35 0 


Iteration 3. 


1. a = min{(10 — (—157/3))/(13/17), (—o0 — 9)/(—3/17), 
(0—245/3)/(—20/17), (co— (—35/6))/(1/17)} = 833/12, p = 


en 
. &g = (3/4, -13/4, —1, -7/4)? column, f = 379/6 — 833/12 = —25/4. 
.S= By 4. 


. max{| — 12/17|}, g =3. 

. Multiply row 3 by —17/12, and then add 15/34, —27/34, and —4/17 times of 
row 3 to rows 1, 2, and 4, respectively (without touching xg column, only the 
third component is replaced by x3 = —1). 


nABwWN 


3/8 5/8  3/2* 
SUR OE) 372 
11712 AA7Ad 3 
es ee 


= 


—13/4 —-7/4 
—oo —7 


™~ 


Iteration 4: 


1. a = min{(10 — 3/4)/(3/2), (—oo — (—13/4))/(—3/2), (2 — (-1))/G/3), 
(—7 — (~7/4))/(-1/3)} = 37/6, p = 1. 

2. ¥3 = (—25/2, 167/18, —137/36, 10)", f = —25/4 — 37/6 = —149/12. 

3. JS =v, 


Dual feasibility is achieved. Convert the end tableau to a conventional dual 
feasible one: delete the xg column, and take the entry at the first row and f column 
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as the pivot to carry out elementary transformations (such that the pivot becomes 
—1). See the resulting tableau below (whose fourth row is now the objective row, 
which was the previous first row): 


x1 x2 x3 X4 x5 X6 X7 f RHS f = —149/12 
1 1372. 1 =12 19 112 =25/2 
1 115/18 —10/9 1/2 —13/18 7/6 167/18 
1 —73/36 2/9 —-1/4 7/36 —37/12| —137/36 
49/12 —2/3 -1/4 5/12 —1} 17/4 10 
u| 9 oo 12 8 10 6 oo 
%|—25/2 —137/36 167/18 0 10 6 0 
|| —oo —7 1 0 0 —5 0 


P1, 02, 3 = 0. The optimal basic solution and optimal value are 


x* = (—25/2, —137/36, 167/18, 0, 10, 6, 0)', 
f* = 4(—25/2) — 5(—137/36) + 2(167/18) = —149/12. 


18.3 Generalized Dual D-Reduced Simplex Algorithm 


Consider the D-reduced bounded-variable problem below: 


In this section, the dual D-reduced simplex algorithm (Sect. 17.2) is generalized 
to solve this problem. A tableau version will be derived first, and then its revised 
version will be formulated. 

The key point is to update the objective row based on the datum row as far as 
possible. 

To trace solution x, three additional rows are listed at the bottom of tableau 
(17.3), 1:2:, 


I NR 
wh 1 
ze} -1 | —-f 
u le sr 
x a it 
l iB i 
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where the associated sets B, N, and R are defined by 
B={j,---,jm—-1}, N=A\B, R= ({l,...,m)}\{r} = {i1,...,im—1}, (18.22) 
r ¢ R; q column is the datum column, satisfying 

@q #9, %q =9. (18.23) 


Let x be the basic solution, defined by (17.7), such that 


xj =ljoruj, JEN, j#aq (18.24) 
kg +Nrin =0, )) o)*j=1. (18.25) 
jen 


If xq enters the basis, x is clearly just the basic solution, associated with the new 
basis in the conventional sense, with objective value 


f = Nin —Zn4t- (18.26) 
Introduce the following column index sets: 
T={jeN, jAq|xj=l}, N={ieN, j #q|xj =uj}. (18.27) 


It is clear that TUM = N\{q}, TN =9. 
The D-reduced tableau, satisfying the following conditions: 


zr = 0, zn <0, (18.28) 


is called dual feasible D-reduced tableau. If components of / and wu are all finite, 
it is always possible to set the nonbasic variables to relevant bounds such that any 
D-reduced tableau is dual feasible. 


Proposition 18.3.1 The objective value f of the dual feasible D-reduced tableau is 
a lower bound of all feasible values. 


Proof Let x’ be an arbitrary feasible solution. Then, it is known from (18.27) that 
srHpflsas. Sy Sr Ss 
which together with (18.28) gives 
xp < Ziel, hat > Zhen, IngitZoxy > ing tien =f. (18.29) 


This indicates that the f is a lower bound on all feasible values if any. oO 
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Provided that x is dual feasible, if, in addition, it satisfies 


lq < Xq Sug; lp < XB < up, (18.30) 


then it is clearly a basic optimal solution. In the other case, a sequence of dual 
feasible tableaus can be generated such that the related objective value increases 
monotonically, until reaching a basic optimal solution or detecting primal infeasi- 
bility. 
Introduce the “boundary violations” as follows: 
Lj — Xj, If x; < lj, 
Pj = uj —x;, If x; > uj, jJe€({i,...,n}. (18.31) 
0, If 1; <SXj < Uj, 


There will be the following two cases to be dealt with: 
(i) Xq violates boundary, 1.e., pg # 0. 
Define the index set 


Ni ={j €T | sign(pg@q) @; < O}U {7 € IT | sign(pya@q) @; > O}, (18.32) 


where sign(f) is the sign function. 
Lemma 18.3.1 Assume pg 4 0. If Ni is empty, there is no feasible solution. 


Proof In view of (18.27), the second expression of (18.25) can be expressed as 


Oq¥q =1—| Sal) + D> ajuj J. (18.33) 
jer jell 


Assume that x’ is a feasible solution, and hence it holds that 


lj <x <u; (18.34) 
and that 
yx, =1—| So ajx, + D0 dix), |. (18.35) 
je. jell 


Consider the case of pg@q > 0. In this case, Nj = % implies 


aj >0,Vjer, 0 <0,VjeN, 
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which along with (18.34) leads to 


S > ajx' + > jx; = YS ajlj + »- @jUj. 


je. jell je. jell 


Then, from the preceding expression, (18.33) and (18.35), it follows that 


Og xe — On @jx), + ‘ms @;x') 


jel jell 
4-() ail) + )> djuj) 
jeD jell 
Dg Xq- 


If @y > 0, pg > 0, the preceding expression gives Xa 


that x’ is a feasible solution. If @q < 0, Pq < 0, on the other hand, it gives x, 
Xq > Ug, also contradicting that x’ is a feasible solution. 

In the case of pg@qz < 0, a contradiction can be derived similarly. Therefore, 
there is no feasible solution. oO 


< Xq < Iq, as contradicts 


> 
ga 


Now assume that N; 4 @. Determine f; and column index gq’ such that 
Bi = leq’ /@y| = min |z;/dj|, 01 = sign(oq%q) 61. (18.36) 
JEM 


If 0; 4 0, the associated D-reduced tableau is said to be dual nondegenerate. 
Add 6; times of the datum row to the bottom row. We might as well denote the 
resulting tableau again by (18.3). Then it is clear that 


Zq = 0104 (18.37) 


and that @, 4 0, 2,’ = 0, where the g’ column is taken as the datum column. 

Let us determine the corresponding solution x and the associated objective value 
f . Since (x, f) and (¥, f ) both are solutions to the system, presented by the new 
tableau, it holds that 


Xp =—Nrin, 3 =—Nréy, (18.38) 
> @jxj =1, @ jx; = 1, (18.39) 
JEN jENn 

f=Zhan—Z1, f= dn — Int. (18.40) 


Denote by a;(R) the jth column of Np for any j € N. 
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Set X, to the violated bound, i.e., 
fq = Eq + py. (18.41) 


Besides Xg and Xq’, let the other nonbasic components of < be equal to those of x 
correspondingly, i.e., 


Rj=xj, GEN GAQ,I#G. (18.42) 
From (18.42) and subtracting the two equalities of (18.39), it follows that 
Wg (Xq — Xq) + Wg! (Kq! — Xq) = 9, 
combining which and (18.41) gives 
Kgl = Xq! — (@q/@q') fq. (18.43) 


From subtracting the two equalities of (18.38) together with (18.41) and (18.42), the 
updating formula follows, i.e., 


Sp = Xp — Pglig(R) — (hq — Xq"diq'(R). (18.44) 


It is easily verified that such a resulting * is a dual feasible solution, associated with 
the new tableau. 

Noting Zg’ = 0, on the other hand, from (18.42), (18.41), and (18.36) together 
with subtracting the two equalities of (18.40), it follows that 


f =f + bq%q = f + lPq%q|Bi = Ff. (18.45) 


Therefore, the objective value strictly increases when the D-reduced tableau is dual 
nondegenerate, i.e., 8; > 0. 

Set gq = q’, and repeat the preceding steps. Under the assumption of dual 
nondegeneracy, the procedure terminates in a finite number of iterations, finally 
satisfying 1, < Xg < ug, or detecting the nonexistence of a feasible solution. Note 
that such an iteration involves no basis change but only updates the objective row 
while maintaining the other part unchanged. 

(2) Xq is within the boundary, i.e., og = 0. 

Determine the row index 


s €argmax {|p;, | f=1,...,m—]}. (18.46) 


If p;, = 0, (18.30) clearly holds, and hence the optimality is achieved. 
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Now assume that p;, # 0. Note that o;, > 0 implies that x;, violates the lower 
bound, and p;, < O implies that it violates the upper bound. Only the following two 
cases arise: 


(i) di,,g #0. 


Taking dj, q as the pivot, convert it to 1, and eliminate the other nonzeros in the 
column by elementary transformations. Thus, j; column leaves from and g column 
enters to the basis. Note that the bottom row remains unchanged since z, = 0. The 
bottom component of j,th column is zero. Since the old @, is nonzero, the new @, 
must be nonzero, and therefore j,th column becomes the new datum column, so set 
q = js. As the solution x is unchanged, it is clear that the new tableau maintains 
dual feasibility, though now the component, associated with the new datum column, 
violates the boundary. Consequently, it becomes case (1) via such an iteration. 


(ii) dj,,q = 0. 

Using the set 

No = {j € Pl sign(o;,) ai,, ; < OJ U {7 € M1 | sign(p;,) ai,, ; > O}, (18.47) 
determine column index q’ and £2 such that 


Ba = |Zq'/4is, q'| = on IZj/ai,, j|, 92 = sign(p;,)B2. (18.48) 
2 


Then, convert Gi, g! to 1, and eliminate the other nonzeros in the column by 
elementary transformations. The basis and nonbasis, associated with the resulting 
tableau, are 


B={ji,..., js—1; OE es eee eee N=N\q' U{js}. (18.49) 


In this case, the associated elementary transformations do not change the gth 
column, which remains to be the datum column. 

Similar to case (1), the x, associated with the resulting tableau, can be obtained 
from (18.38) and (18.40) as follows. Set the nonbasic component Lae to the violated 
bound, i.e., 


Kj, =Xj, + Pjy- (18.50) 


Let all nonbasic components of < be the same as those of x, except for the g and 
js-indexed ones. Thus, from (18.39), it follows that 


Og (Ag = Hg) +O, A, =F, )—0. 
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Thus, we have the following updating formula: 
Xq = Xq — (@j,/@q)p;,- (18.51) 
Then, by (18.38), it holds that 
XB = Xp — pj,(aj,(R) — (Xq — Xq)aq(R)). (18.52) 


The new tableau is dual feasible for x. 
From that z;, = 62 holds for the new tableau, (18.40) and (18.48), it follows that 


f=ft+o,2;, =f +0; =f +lo;,lo>= f, (18.53) 


which implies that the objective value does not decrease and strictly increases when 
the D-reduced tableau is dual nondegenerate. 

The overall steps can be summarized to the following algorithm, where the 
objective value is not updated by (18.53) iteration by iteration but is computed at 
the end of the algorithm instead. 


Algorithm 18.3.1 (Generalized Dual D-Reduced Simplex Algorithm: Tableau 
Form) Initial: dual feasible D-reduced tableau of form (18.3), qth column is the 
datum column, associated with solution x. This algorithm solves the D-reduced 
bounded-variable problem (18.3). 


. Go to step 7 if pq defined by (18.31) is 0. 

. Stop if N, defined by (18.32) is empty. 

. Determine B, and column index q' by (18.36). 

. If By #0, add fp, times of the datum row to the bottom row. 

Update Xq, Xq', and Xg by (18.41), (18.43), and (18.44). 

Set q = q', and go to step 1. 

Determine s by (18.46). 

. Stop with f = c'x if pj, = 0. 

. Go to step 12 if aj, g = 0. 

. Convert aj,.q to 1, and eliminate the other nonzeros in the column by 
elementary transformations. 

. Setg = js, and go to step 2. 

. Determine column index q' by (18.48). 

. Convert aj,, q’ to 1, and eliminate the other nonzeros in the column by 
elementary transformations. 

. Update x;,, xq, and xp by (18.50), (18.51), and (18.52). 

. Go to step 1. 
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Theorem 18.3.2 Algorithm 18.3.1 creates a sequence of basic feasible solutions. 
Under the dual nondegeneracy assumption, it terminates either at 


(i) Step 2, detecting infeasibility of the problem, or at 
(ii) Step 8, giving a basic optimal solution 


Proof The validity comes from Lemma 18.3.1 and the analysis preceding Algo- 
rithm 18.3.1. Oo 


Example 18.3.1 Solve the following problem by Algorithm 18.3.1: 


min f = 2x; — 4x2 — 7x3 + 5x4, 


s.t. —3x, +x2 +3x3 +5x4 +%5 = 5, 
—6x1 +2x2 —4x3 —4x4 +x6 = —20, 
4x, —x2 —2x3 +3x4 +x7 = —8, 


—10<x,<10, -8<x.<20, Il<x< 15, 
—10<x4< 17, O<x5<15, —-1< x6€ 12, 
—8 <x7 < 15. 


Answer Initial tableau: 


Xx] x2 x3 X4 x5 X6 x7 RHS 
—3 1 3 5 1 5 
—6 2 —4 —4 1 —20* 

4 -1 —2 3 1 —8 
2 —4 —7 5 


Iteration 1: Convert the preceding to a D-reduced tableau. 
Take xy = (—10(-), 20, 15, -10;_))? (NW = {1, 2, 3, 4}). 
Xp = b — Nxy = (—40, —100, 112)" (B = {5, 6, 7}). 
ps5 = 0— (—40) = 40,96 = (—1) — (—100) = 99, 
p7 = 15 — 112 = —97. max{| — 40], | — 99], |97|} = 99, p =2. 
Enter the right-hand side to the basis: multiply row 2 by 1/20, and then add 5 
and —8 times of row 2 to rows | and 3, respectively: 


3/10  —1/10 1/5 
32/5 ~9/5 ~2/5 


~ es 
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The second row is the datum row, and x6 column is the datum column. 

Xp = (—40, 112)'(B = {5, 7}). 

ky = (-10-), 20M, 15, —10(_), -100)? (NW = {1, 2, 3, 4, 6}). f = —255. 
The resulting tableau is then dual feasible. 

Call Algorithm 18.3.1:r = 2, q =6. 
Iteration 2: 


1. p6 = —1 — (— 100) = 99 > 0. 
. sign(p6a2,6) = —1, Ni = {1,2, 45 4G. 
- By = min{|2/(3/10)|, | — 4/(—1/10)], |5/./5)|} = 20/3, 
q' =1, 6 = —1(20/3) = —20/3. 
4. Add —20/3 times of row 2 to row 4. 
. X6 = —1004 99 = —1. x) = —10 — ((—1/20)/(3/10))99 = 13/2. 
Xp = (—40, 112)T—99(1/4, —2/5)™ — (13/2—(—10))(—9/2, 32/5)? 
= (19/2, 46)", B = {5, 7}. 


Wd 


Nn 


6.qg=1. 


—9/2 479 D 4 
3/10  —1/10 1/5 1/5 
32/5* —9/5 2975 23/5 


-10/3. —25/3 11/3 


~ 4S 


Iteration 3: 


1. op, = 0. 

7. p5 = 0, p7 = 15 — (46) = —31. 13 = 3, 3 = 7. 

9. a3.) = 32/5 £0. 
10. Multiply row 3 by 5/32, and then add 9/2 and —3/10 times of row 3 to rows 1 


and 2, respectively. 
ll. g=7. 
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15/64 55/32 463/64 1 -1/32 45/64 
—1/64 7/32 1/64 =1/32. -=3/64 
9/32. 1/16 23/32 —1/16 5/32 


10/3 25/3 11/3 1/3 
20 15 17 12 15 
=10 


~ as 


Iteration 4: 


_ 


OW 


. 7 = 15 — 46 = —30 < 0. 
: sign(07a2,7) => 1, N = 13% 4, 6} FH 0. 
- By = min{|(—25/3)/(7/32)|, |.1/3)/(—1/64)|, |1/3)/(—1/32)|} = 32/3, 


gq’ =6, 6 = —(-1)(32/2) = 32/3. 


. Add 32/3 times of row 2 to row 4. 
. X7 = 464 (—31) = 15; X% = —1 — ((—3/64)/(—1/32))(—31) = 91/2; 


Xp = (19/2, 13/2)'—(—31) (45/64, 5/32)'—(91/2—(—1))(— 1/32, —1/16)" 
= (131/4,57/4)', B = {5, 1}. 


q = 6. 
15/64 55/32 463/64 1 —1/32 45/64 
—1/64 7/32 1/64 1/32 —3/64 
1 O82 = T/16:- ° 93/32 —1/16 5/32 
u 10 20 15 17 15 12 15 
Fd 57/4 20 15 -10 131/4 91/2 15 
l —10 —8 1 —10 0 -1 —8 


Iteration 5: 


nABRWN eS 


. 06 = 12—91/2 = —67/2 < 0. 

. sign(ped2,6) = 1, Ni = {3,4} 4 VY. 

- Bi = min{|(—6)/(7/32)], |(7/2)/(—1/64)|} = 192/7, q’ = 3, 0 = 192/7. 
. Add 192/7 times of row 2 to row 4. 

. X6 = 91/24 (—67/2) = 12; 


¥3 = 15 — ((—1/32)/(7/32))(—67/2) = 143/14; 
¥p = (131/4, 57/4)" —(—67/2)(—1/32, —1/16)?— (143/14 — (15)) (55/32, 
—1/16)" 
= (559/14, 83/7)', B = {5, 1}. 
q =3. 
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X3 X4 X5 X6 
15/64 55/32* 463/64 1 -1/32 45/64 
—1/64 7/32, 1/64 1/32 —3/64 
9/32. —1/16 23/32 —1/16 5/32 


—55/14 43/14 =6/7 —- 25/44 
15 17 15 
20 143/14. —10 559/14 
1 0 


Iteration 6: 


1. 93 = 0. 

7. p5 = 15 — 559/14 = —349/14, op; = 10 — 83/7 = -13/7. i) = 1, fj) =5S. 
9. a13 = 55/3240. 
10. Add —7/32, 1/16 times of row | to rows 2,3, respectively: 
ll. g=5S. 


X4 x5 X6 
463/110 32/55 —1/55 9/22 
103/110 7/55 —3/110  —3/22 
54/55 2/55 —7/110 2/11 


—55/14 43/14 =6/7) <=25/id 
20 17 5 


10 15 15 12 1 
83/7 20 143/14 —10 559/14 12 
—10 —8 1 —10 0 —1 


Iteration 7: 


. ps5 = 15 — 559/14 < 0. 
. sign(psa2,5) = 1, Ni = {4} 4G. 
. Br = min{|(43/14)/(103/110)|} = 2365/721, gq’ = 4, 6; = 2365/721. 
. Add 2365/721 times of row 2 to row 4. 
. X5 = 559/14 + (—349/14) = 15; 

x4 = —10— ((—7/55)/(—103/110))(—349/14) = —681/103; 

XB = (143/14, 83/7)? — (—349/14)(32/55, 2/55)? 

—(—681/103 — (—10))(463/110, 54/55)? 
= (1077/103, 972/103)", B= {3,1}. 

6.q =4. 


aABWN eRe 
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x] x2 X3 x4 x5. X6 “x7 | RHS 
3/22 1 463/110 32/55 —1/55 9/22 
—1/22 —103/110 —7/55 —3/110 —3/22 1 
1 3/11 54/55 2/55 —7/110 2/11 
—420/103 —43/103 —195/206 —230/103 | 3575/103 
u 10 20 15 17 15 12 15 
xX | 972/103 20 1077/103 —681/103 15 12 15 
Z| -—10 -8 1 —10 0 -1 -8 


Iteration 8: 


1. pa = 0. 
7. 03 = p1 = 0. The optimal solution and optimal value are 


x* = (972/103, 20, 1077/103, —681/103, 15, 12, 15), 
f* = (2, —4, -7, 5)(972/103, 20, 1077/103, 681/103)’ = —17240/103. 


Based on the equivalence between the D-reduced tableau (17.3) and the revised 
tableau (17.19), the revised version of Algorithm 18.3.1 can be stated as follows. 


Algorithm 18.3.2 (Dual D-Reduced Simplex Algorithm) Initial: B, R, N, 
R iN = cw — NEBR'cp. x, and f, satisfying (18.24) and (18.28). This 
algorithm solves the D-reduced bounded-variable problem (18.3). 


1. Go to step 8 if pq, determined by (18.31), is equal to 0. 
2. Compute oy = Nl em — NE (BR: Bem). 
3. Stop if N, ={j €T | sign(pga@,)@; < O}U{j € TL | sign(pga@,)o; > 0} =H 
(infeasible problem). 
4, Determine 6, and column index q' such that By = |Zq’/@q’\| = 
minjen, |2;/@jl. 01 = sign(o4@q)B. 
5. If 0; #0, update zy =Zn + O10n. 
6. Update Xg, Xq’, and xg by (18.41), (18.43), and (18.44). 
7. Set q = q', and go to step 1. 
&. Determine s by (18.46). 
9. If pj, = 0, compute f =", and stop (optimality achieved). 
10. Compute oy = N}BR es. 
11. Go to step 15 if og = 9. 
12. Compute Gg(R) = Bp dq(R). 
13. Update By’ by (17.20). 
14, Set q = js, and go to step 2. 
15. Determine column index q' such that |2q'/oq'| = min jen, |Z; /o;)|.- 
16. Compute Gq/(R) = Br'dq/(R). 
17. Update By’ by (17.20). 
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18. Update xj, ,Xg', and xp by (18.50), (18.51), and (18.52). 
19. Update (B, N) by exchanging j; and q’. 
20. Go to step 1. 


Chapter 19 ®) 
Deficient-Basis Method Ghost for 


As was mentioned, the assumption of nondegeneracy is entirely contrary to reality. 
When the simplex method is used to solve large-scale sparse problems, high 
degeneracy occurs frequently. In practice, the real problem caused by degeneracy 
is not cycling but stalling, in which case, the algorithm can be so time-consuming 
that an unacceptably large number of iterations stay at a degenerate vertex before 
exiting it. 

In this chapter, the basis is generalized to allow deficiency to overcome this 
shortcoming. This brings up the possibility to take advantage of high degeneracy to 
reduce computational work significantly. The new algorithms cannot be classified 
to the simplex category, because each associated iteration no longer necessarily 
corresponds to a simplex. 

In the next section, the concept of the deficient basis is introduced first. In the 
subsequent two sections, the deficient-basis algorithms in tableau form and revised 
form are derived in turn, together with the associated computational results reported. 
In the section after, we address some special implementation issues, such as the 
initial basis, the LU updating, and the Phase-I procedure using the single-artificial- 
variable scheme. In the final section, we incorporate the deficient basis into the 
reduced simplex algorithm together with the Phase-I procedure using the most- 
obtuse-angle rule. 

Consider the standard LP problem but assume that the system Ax = b is 
consistent, or in other words, the column space of A includes J, i.e., 


b € range(B). 
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19.1 Concept of Deficient Basis 


The basis has been the center of the simplex method for many years. Since it is 
a submatrix consisting of m linearly independent columns of A, however, it is 
unavoidable for some basic components of the solution to vanish, whenever the 
right-hand side belongs to a proper subset of range(B). To cope with this difficulty, 
we use the following generalized basis instead. 


Definition 19.1.1 Deficient basis is a nonsingular square submatrix of A such that 
the subspace spanned by the associated columns of A includes b. 


It is clear that there exists such a basis under the assumption of b € range(A), 
and that a deficient basis is a normal basis if its order equals m. 

The columns/rows associated with a deficient basis are said to be basic, and the 
other columns/rows are nonbasic. Variables (components) are said to be basic and 
nonbasic, respectively. 

Let k be the order of a basis. Denote by B and R the index sets of basic columns 
and rows, respectively by 


B={ji,..., je}, and R= {iy,...,,x}. (19.1) 
Denote the associated nonbasic column and row index sets, respectively by 
N=A\B, and R’={l,...,m}\R. (19.2) 


Once B, R are given, so are N, R’. Note that R’ is empty when k = m whereas N is 
always nonempty. Although | < k < m is usually assumed, it is sometimes useful 
and easy to include the case k = 0, in which B and R are both empty. 

Without confusion, the preceding set notation will also be utilized to denote 
the submatrices or vectors consisting of the associated columns and rows; e.g., the 
submatrix consisting of basic columns is still denoted by B, as well as the following: 

Br € RE**: basis, consisting of entries in basic columns and rows. 

Br € R"—**: submatrix consisting of entries in basic columns and nonbasic 
rows. 

Nr € RkX—4; submatrix consisting of entries in nonbasic columns and basic 
rows. 

Ne € RO—Y*x@-®: submatrix consisting of entries in nonbasic columns and 
rows. 

br € Ré: subvector consisting of basic components (in basic rows) of the right- 
hand side. 

br € R”—*: subvector consisting of nonbasic components (in nonbasic rows) 
of the right-hand side. 

ag(R) € Ré: the q-indexed column of Np. 

dq(R’) € R”-*: the q-indexed column of Np. 
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Since b € range(B), an initial tableau of the standard LP problem, say 


xp xy  f | RHS 


Br NR br 

19.3 
Br Nr DR es 
Ch Gre —1 


can be converted to an equivalent form by elementary transformations with row and 
columns exchanges, i.e., 


[Nr br (19.4) 
Nr 
zL -1| —f 


The preceding is referred to as deficient-basis tableau, where the unit matrix [ 
corresponds to basis Br, featured by basic columns and rows. 
ay is the reduced cost(s). The solution to Ax = b, given by 


Xp=br, xn =0, 


is called basic solution. It is always possible to expand a deficient basis to a full one, 
associated with the same solution, by entering some nonbasic columns. Note that 
the j,-indexed basic variable xj, is in the i;-indexed row of the tableau. Therefore, 
it holds that 


x; =b;,, t=1,...,k. 


The solution is feasible if xg > O, and optimal if, in addition, the associated 
objective value attains the minimum value over the feasible region. The tableau 
is said to be feasible or optimal, respectively. 

There are results similar to those in the conventional simplex context; for 
instance, we have the following. 


Lemma 19.1.1 Assume that the deficient-basis tableau (19.4) is feasible. If its 
reduced costs are all nonnegative, then it gives a pair of primal and dual basic 
optimal solutions. 


Proof It is clear that the tableau gives a pair of complementary primal and dual 
solutions, which are primal and dual feasible respectively, satisfying the optimality 
conditions. Oo 
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19.2 Deficient-Basis Algorithm: Tableau Form 


Let us now discuss how to transform a feasible deficient-basis tableau to a new one 
while maintaining feasibility, and therefore derive a tableau version of the so-called 
deficient-basis algorithm. 

Assume that the current deficient-basis tableau of form (19.4) is feasible (b R= 
0) but zy Z O. We select a pivot column index g such that 7, < 0, e.g., by the 
counterpart of Dantzig conventional rule below: 


q € arg min{z; | j € N}. (19.5) 


Thus, it holds that z, < 0. 7 _ 

Denote the q-indexed columns of Nr and Nr’ by ag(R) and ag (R’), respectively. 

The determination of a pivot row index depends on whether column dg (R’) 
vanishes, i.e., whether a, belongs to the column space of B. There are the following 
two cases arising: 

(i) Rank-increasing iteration: R’ 4 @ and ag (R’) £0. 

In this case, it is not allowed for x, to be taken on any nonzero value while 
keeping the other nonbasic components of the current solution unchanged, as such 
doing violates some of the equality constraints, associated with nonbasic rows. To 
overcome the difficulty, we determine the row index 


p € argmax |d;,q|. (19.6) 
ie R’ 


Then take dpq as a pivot to carry out elementary transformations, so that the q- 
indexed nonbasic column and p-index nonbasic row both become basic while the 
basic columns and rows remain basic. Thus, the rank of the basis increases by 1, and 
the iterations are complete. It is noted that the right-hand side remains unchanged. 

(ii) Rank-remaining iteration: R’ = % or ag (R’) = 0. 

Increase x, from zero to decrease the objective value while keeping the other 
nonbasic components of the current solution feasible. In the following case, such a 
decrease will be unlimited. 


Lemma 19.2.1 Assume that the deficient-basis tableau is feasible and that R' = G 
or Gg(R') = 0. If Zq < 0 and ag(R) < 0, the original problem is lower unbounded. 


Proof The proof is an analogue to that of Lemma 3.1.2. oO 
If, otherwise, the expression a,(R) ¢ 0 is not satisfied, then there is a blocking 
variable, associated with a and s such that 


a= bi, /ai,, q = min{bi, /di,,¢ dig >0, t=1,...,k} = 0, (19.7) 


where stepsize a will be used to update the feasible basic solution. Taking qj, g as the 
pivot, carry out according elementary transformations. Consequently, the js-index 
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column leaves and qg-index column enters the basis, while the basic rows remain 
basic. So, the rank of the basis remains, and an iteration is complete. It is noted such 
doing does not touch nonbasic rows at all (including entries in the right-hand side). 

As Z, is eliminated, both cases change components of the dual solution, located at 
the bottom row. Since b;, = 0, a rank-increasing iteration does not change tableau’s 
right-hand side, hence the solution and objective value. On the other hand, a rank- 
remaining iteration changes these quantities, in general; in fact, it is known that the 
objective value becomes 


f=ftay<f, 


where the right-most inequality results from (19.7) and z; < 0. Therefore, the 
objective value does not increase, and in case of nondegeneracy (br > 0), @ is 
positive, hence the objective value strictly decreases. 

The overall procedure can be summarized into the following algorithm. 


Algorithm 19.2.1 (Deficient-Basis Algorithm: Tableau Form) Initial: feasible 
tableau of form (19.4) with B, N, R, R’, 1 < k < m. This algorithm solves 
the standard LP problem. 


1, Determine column index q € arg min jen Z;. 
2. Stop if Zq = 0 (optimality achieved). 
3. Go to step 7 if R' = 9 or ag(R’) = 0. 
4. Determine p € arg maxje Rr’ |Gj q|. 
5. Convert ap q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
6. Setk =k +1, bring q from N to B and p from R' to R, and go to step 1. 
7. Stop if ag(R) < 0. 
8. Determine s by (19.7). 
9. Convert aj;,,qg to 1, and eliminate the other nonzeros in the column by 
elementary 
transformations. 
10. Update (B, N) by exchanging js and q. 
11. Go to step 1. 


Theorem 19.2.2 Assume that rank-remaining iterations are all nondegenerate. 
Algorithm 19.2.1 terminates either at 


(i) step 2, giving an optimal basic solution; or at 
(ii) step 7, detecting lower unboundedness. 


Proof The proof on termination of Algorithm 19.2.1 is similar to that in the 
conventional case. The meanings of its exits come from Lemmas 19.1.1 and 19.2.1 
together with the related discussions preceding Algorithm 19.2.1. Oo 
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Example 19.2.1 Solve the following problem by Algorithm 19.2.1: 


min f = —2x; +3x2 —4x3 +x4 +2x7 —3x8, 
S.t. Xx] x2 x3 X4 
2x1 x2 x3 X4 
xy —2x2 —x4 
al +x2 +x4 
2x1 X20 +rX4 
xj 20, jHl,...,8. 


tX5 


bX —2xg 

+x6 —Xx7 +XxXg 
—3xg 

+x 

—Xxg 


Answer k = 2,B = {5,6}, R = {1,2}, N = {1,2,3,4,7, 8}, R’ 
The initial feasible deficient-basis tableau: 


Iteration 1: 


a, (R’) = 0. 
a,(R) £0. 


Se ot 


x2 x3 x4 
1 1 -1 
-1 1 1 
—2 -1 
1 1 
-1 1 
3. -4 1 


x2. 1X3 x4 
2 —2 
-1 1 1 
—2 -1 
1 1 
-1 —2 


minf{2/1,1/1}=1, s=2, i, =2. 
add —1, 4 times of row 2 to rows 1, 6, respectively. 
B= {5,3}, N = {1, 2,4, 6, 7, 8}. 


min{—2, 3, —4, 1,2, -3} = —4 <0, g =3. 


= 19,4, 5}. 
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Iteration 2: 


1. min{6, —1,5,4, —2, 1} =-2 <0, g=7. 

3. dg(R’) = 0. 

7. Gg(R) £0. 

8. min{1/2} = 1/2, s=1, is =1. 

9. Multiply row 2 by 1/2, then add 1, 2 times of row 2 to rows 2, 6, respectively. 
10. B = {7,3}, N = {1, 2,4, 5, 6, 8}. 


Xx] x2 X3 X4 X5 X6 x7 RHS 
= | Sit iy. aye 1/2 
1/2 1 1/2 1/2 3/2 
1 —2 -1 
1 1 1 
2 = —2 
3 1 3 i 3 5 


Iteration 3: 


. min{3, 1,3, 1,3, -2} = -—2 <0, g =8. 

. g(R’) # 0: 

. max{| — 3], |1],| —1]}=3, p =3. 

. Multiply row 3 by —1/3, then add —1,1,2 times of row 3 to rows 4,5, 6, 
respectively. 

6.k =3, B ={7,3, 8}, R= {1, 2,3}, N = {1,2,4,5, 6}, R’ = {4, 5}. 


an BR We 


xX] XQ 4X3 x4 x5 X67 
—2 2 -1/2 1/2 -1/2 1 
1/3 1/3 1 1/6 1/2 1/2 

-1/3 2/3 1/3 

—2/3 1/3 2/3 
5/3 —-1/3 —5/3 
7/3 7/3 11/3 1 3 


Iteration 4: 


1. min{7/3, 7/3, 11/3, 1, 3} > 0. 
2. The optimal basic solution and optimal value are: 


x* = (0, 0,3/2,0,0,0,1/2,0)7,  f*=—5. 
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19.3 Deficient-Basis Algorithm 


In this section, we derive the revised version of Algorithm 19.2.1. 

The deficient basis characterizes an iteration, just as the normal basis in 
the conventional simplex context. All entries in a deficient-basis tableau can be 
expressed using the inverse of the deficient-basis matrix. 

Let Br be the basis matrix associated with sets B, R, and let Be be its inverse. 
Then, it is known that 


br = br — Br Bp br =0. 
Thus, the revised tableau, corresponding to the deficient-basis tableau (19.4), is 


XB XN f RHS 

I Bp Nr Br br 
NR’ — Br BR'Nr 
cl. —chB, Nr —1|—chB, br 


(19.8) 


Let g € N be the pivot column index selected by (19.5). The following tableau 
offers correspondence between wanted quantities in the preceding tableau and 
tableau (19.4) (see Table 19.1): 

As for updating the inverse Be of the deficient basis, the following two types of 
iterations should be treated separately: 

(i) Rank-increasing iteration: R’ 4 @ and ag(R’) = ag(R’)— Br’ Be 0g (R) £0. 

Assume that pivot row index p € R’ was determined by (19.6). The new basis 
matrix (increasing rank by 1) is of the form 


~ BR \ag(R)\ k 
Ba = |= 19.9 
R (4 Gi, 1 ( ) 
k 1 


where e,B denotes the pth row of B, ap), denotes the pth component of ag, and 
dg (R) corresponds to basic components of ag. It is easy to verify that the inverse is 


Table 19.1 Equivalence between the associated quantities 


Quantity Tableau Relation Revised tableau 
Objective row ZN = ch _ ch By Nr 

Basic components of pivot column aq(R) — Br I dg(R) 

Nonbasic components of pivot column aq (R’) a g(R') — Br Br ag (R) 
Basic components of the right-hand side br ; = Bab R 


Nonbasic entries of the pth basic row enNr = e, By Nr 
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of the following form: 


ieee ( UY | ‘ (19.10) 


where 
h' = eB, 
t = 1/(apq — h'ag(R)), 
v = —Ta,(R), 
d' =—th'By', 
-1 T p-l 
U=B, —vh Br. 
(ii) Rank-remaining iteration: R’ = @ or dg(R’) = ag(R’) — Br Bp dq (R) = 0. 
Essentially, the updating, in this case, is the same as in the conventional simplex 


context, only Be is now of order less than m, in general. Assume that s € 
{1,..., k} was determined by (19.7). The updating formula is 


B,!| = EB, (19.11) 
where 


1 — (4g (R))1/(dq(R))s 


— (4g (R))s—1/ (ag (R))s 
Es = 1/(dg(R))s 
—(dq(R))s+1/(dq(R))s 


— (4g (R))k/(aq(R))s 1 


Based on Table 19.1, Algorithm 19.2.1 can be revised as follows. 


Algorithm 19.3.1 (Deficient-Basis Algorithm) Initial: B, R, 1 <k <m, Bee 
XB= Bp br > 0. This algorithm solves the standard LP problem. 


1. Compute zy = cn — NiTy, where y = Bai ce. 

. Determine pivot column index q € arg min jen Z;. 

. IfZq = 0, compute f= olan, and stop (optimality achieved). 
. Compute ag(R) = Br dq(R). 

. Go to step 9 if R’ = Y or Gg(R’) = ag(R’) — Brag (R) = 0. 


MAR WH 
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6. Determine row index p € arg maxjeR’ |i q|. 

7. update By’ by (19.10). 

8. Setk =k +1, bring q from N to B and p from R' to R, and go to step 1. 

9. Stop if dg(R) < 0 (lower unbounded problem). 

0. Determine s and a such that a = x;j,/Gi,,g = min{x;,/di,,q | Gig > 9, t = 
1, sa25.k}: 

11. Ifa #0, set xg = a and update xg = XB — aag(R). 

12. Update By’ by (19.11). 

13. Update (B, N) by exchanging js and q. 

14. Go to step 1. 


Example 19.3.1 Solve Example 19.2.1 by Algorithm 19.3.1. 


Answer Initial: k = 2, B = {5,6}, R = {1,2}. N = {1,2,3,4,7,8}, RF’ = 
{3,4,5}. Be = 1. Xp = (2,1)' > 0. 
Iteration 1: 


1. Zw = (—2, 3, —4, 1, 2, -3)7. 
2. min{—2, 3, —4, 


9. Gg(R) £0. 
10.a@= min{2/1, 1/1} = 1, s= 2; ls = 2. 
11. %3=1, ¥g = (2,1)? — 101, 1)? = (1, 0)". 


ema (NC)= C3) 


13. B= {5,3}, N = {1, 2,4, 6, 7, 8}. 


Iteration 2: 


1. y=(,-4), 
Zn = (—2,3,1,0,2,—3)' — (-8,4,-4,-4,4,-47 = (,-1,5,4, 
—2,1)7, 


2. minf6,—1;5,4)=2, } = —2 <0, g=7. 


a= (1) (1) =(): 


0 00 F 
5. a(R’) =| 0} —|[ 00 (7) 40. 
0 00 


9. Gg(R) £0. 
10. w = min{1/2} = 1/2, s=1, i, =1. 
11. *3 = 1/2, xg = (1, 1)? — (1/2)(2, -1)" = (0, 3/2)". 
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12. Bol = 1/20) (P=1\. (1/72=1/2)\: 
1724 1 1/2 1/2 
13. B = {7,3}, N = {1,2,4,5, 6, 8}. 
Iteration 3: 
1. y = (=1,=3)', 
Zw = (—2,3,1,0,0, —3)? — (—5, 2, —2,-1, -3,-1)? = (3,1,3,1, 

3, —2)!. 

2. min{3, 1,3, 1,3, -2}= —2 <0, g =8. 


ae f WO? (ey. fe 
+ 44®= (Gn ta) (1) = (ia) 


4 00 3 
5. a,(R')= | 1]—| 00 (aa) 1) 40. 
1 00 <i 


6. max{| — 3], [1], |—1]}=3, s =3, is =3. 
7. t = (—3 — (0, 0)(—3/2, —1/2)')-! = -1/3. 
v = —(—1/3)(—3/2, —1/2)? = (—1/2, -1/6)', d™ = (0,0). 


_ (2-1/2) _ (1/2 _ (1/2 -1/2 
v= (1, iD) (146) = (ip De: 


2-1/2 =12 
B,' = 1/2 1/2-1/6 
0 0-1/3 


> 


8.k=3, B={7,3,8}, R={1,2,3}, Bp = Ba’, X= 0. 
Iteration 4: 


1.9 = (= =3, 2/3)", 
Zy =(—2, 3, 1, 0, 0) —(—13/3, 2/3, —8/3, —1, —3)7 =(7/3, 7/3, 11/3, 1, 3)7 0. 
3. The optimal basic solution and optimal value: 


x* = (0,0,3/2,0,0,0,1/2,0)", — f* = (—4, 2)(3/2, 1/2)" = -S, 


which coincides with the result in Example 19.2.1. 


It is seen that the deficient-basis algorithm solves two linear systems in each 
iteration, just as the simplex algorithm; but the systems are smaller, especially for 
large-scale and highly degenerate problems. Also, the deficient basis used appears 
better conditioned. Its advantages are conformed by computational experiments, as 
reported in the following subsection. 


Note In the original work (Pan, 1998-2008), the basis is generalized as a rectangle 
submatrix B, whose column space includes b. It is then modified to a square 
submatrix. Unlike the simplex algorithm, therefore, the solution process of the 
deficient-basis algorithm cannot be interpreted as a movement from simplex to 
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simplex. Since the A is not of full row rank in general, there is no correspondence 
between the deficient bases and vertices. 


As for the bounded-variable problem (25.1), the concept of basis can be generalized 
as follows. 


Definition 19.3.1 Let Br be the nonsingular square submatrix consisting of entries 
indexed by B and R, and let xy be some fixed values of the nonbasic variables 
indexed by N. If the range space of B includes b — Nxy, the submatrix Br is a 
basis. 


Based on the preceding definition, it is not difficult to generalize the deficient- 
basis algorithm for solving the bounded-variable problems. 


19.3.1 Computational Results 


We report computational results to demonstrate the performance of the deficient- 
basis simplex algorithm Pan (2008). 

The experiments were conducted on a Pentium II 550E PC, with Windows 98 
operating system, 256MB RAM, and about 16 decimal point precision. Visual 
Fortran 5.0 compiler was used. 

There were two codes involved in the comparison: 


1. MINOS 5.3. 
2. PDBSA 1.0:is based on an algorithm close to Algorithm 19.3.1 


The second code is implemented within MINOS 5.3; the changes are limited on 
its core part. 

Total 65 standard test problems without bounds and ranges sections were tested, 
which are classified into two groups: 

The first group includes 50 Netlib test problems (Appendix C: Table C.4: AFTRO- 
WOODW), classified into 3 sets according to increasing m+n: Small, Medium, and 
Large. The Small includes 20 problems, Medium and Large include 15 each. 

The second group contains 15 larger problems, including 8 Kennington problems 
(Table C.5), 4 BPMPD (Table C.6), and the 3 largest Netlib problems (Table C.4: 
QAP12, MAROS-R7, STOCFOR3). 

Iteration and time ratios of MINOS 5.3 to PDBSA 1.0 are listed in Table 19.2 
where the end column gives degenerate iterations percentage ratio. The largest 3 
Netlib problems are not included, as MINOS 5.3 failed to solve them. 

At first, it is seen from Table 19.2 that the new code outperformed MINOS 5.3 
overall: 

The time ratios for the first and the second groups are 1.41 and 2.06, respectively. 

It seems that the new code has greater advantages in dealing with large-scale 
problems. 
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Table 19.2 Iteration and 
time ratios of MINOS 5.3 to 
PDBSA 1.0 


Problem Iterations | Time | % Degen 


Small (20) _| 1.09 1.50 | 1.06 
Medium (15) | 0.96 1.30 | 0.85 
Large (15) | 1.08 1.41 | 0.87 
Average (50) | 1.07 1.41 | 0.93 
Kennington(8) | 1.51 2.07 | 0.61 
BPMPD(4) | 1.64 2.05 | 0.63 
Average (12) | 1.52 2.06 | 0.63 


The new code appeared to be more stable and reliable, as it solved all the 65 
test problems, including NSCT2, MAROS-R7, and STOCFOR3, which MINOS 5.3 
failed to solve. 

It is noticeable that the time ratios are significantly larger than iteration ratios. 
This is not surprising since the systems solved in each iteration by the new code are 
small, compare with those solved by MINOS 5.3. 

Finally, it is seen from the last column that except for the smallest 20 problems, 
degenerate iterations ratios are less than 1, that is, the proportion of degenerate 
iterations yielded by the new code is larger than that by MINOS 5.3. Even so, the 
new code was still an unambiguous winner. Forrest and Goldfarb (1992) are right 
that the efficiency of an LP solver does not depend on the proportion of degenerate 
iterations. 


19.4 On Implementation 


As a variant of the classical simplex algorithm, the implementation issues of the 
deficient-basis algorithm, such as the LU factorization of the basis matrix, Lo! 
stored in factors form, LU update in the rank-remaining iteration, etc., are analogous 
to those, presented in Part I. Besides all, one should pay close attention to taking 
advantage of basis deficiency to reduce computational work. In this section, only 
the following three themes are addressed. 


19.4.1 Initial Basis 


It is noticeable that 
rank (A) = rank (A | b) < m. 


The LU factorization of an initial basis matrix can be created by a crash procedure 
like that presented in Part I. In addition to balancing sparsity and stability, however, 
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the resulting initial basis should be of order as low as possible. To this end, some 
variant of the Markowitz Rule would be utilized. 

Initially, set B = %, R = &%. Suppose, for some k, Gauss elimination has 
been carried out on the matrix (A | b) such that some k columns (excluding b) 
of the latter are upper triangular; in other words, k basic columns and rows have 
been determined, and the according Br is an upper triangular k-order submatrix of 
(A | dD). 

If k =m, orif k < m but the following row index set 


I' = {i € R’ | |b;| > €0} (19.12) 
is empty, where €g > 0 is a threshold for numerical 0, then the LU factorization of 


an k-order basis matrix has already been obtained. Otherwise, the following rule is 
utilized to determine the (k + 1)th basic column and row. 


Rule 19.4.1. Given a constant o withO < o < 1. Select the column index q and 
row index p such that 


(rp—1) x (cg—1) = min{(7;—-1) x (cj) | |@ij| > 09, CE T', FE NJ, (19.13) 


where r; denotes the number of the nonzeros in the ith row and c; the number of the 
nonzeros in the jth column, and 


6 = max{|a;j| |i ¢ 1’, j € N}. (19.14) 
Note that the preceding rule fails if 
{G, j) € I x N | |aij| => 06} = 9. 


In this case, Ax = b may be regarded numerically inconsistent. 


19.4.2. LU Updating in Rank-Increasing Iteration 


Updating LU factors in a rank-remaining iteration is essentially the same as that 
in the conventional simplex context. In contrast, updating LU factors in a rank- 
increasing iteration is simpler. 

Let L~!Br = U be the LU factorization of Br. From (19.9) it follows that 


i ree pe aR) 
ee: ei B ding , 


which is upper triangular, except for the bottom row. Premultiply a series of 
Gauss matrices G;,..., Gx to eliminate entries of the bottom row (except for the 
diagonal), it becomes an upper triangular matrix U. The LU factorization of the new 
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basis matrix is then 


7 ak 7 7 —l 
Lo Bee. i= G,...61(4 7): 


19.4.3. Phase-I: Single-Artificial-Variable 


Standard Phase-I methods presented in Chap. 14 are applicable (or after slightly 
modified) in the deficient-basis context. It is also possible to design new Phase-I 
methods by taking advantage of the deficient basis, like in the Phase-I algorithm 
introduced in this section (Pans 2001). 

Introducing an artificial variable f, associated with column the same as the right- 
hand side. The resulting auxiliary problem is then 


min f 


19.15 
st. Ax+bxn41=b, x, f >0: ( ) 


which has the initial tableau below: 


Take the x,4, column as the first basic column. Assuming b, ¥ 0, take the rth 
row as the first basic row, i.e., 


B={n+1}, R= fr}. 
Thus, 
B={n+l}, N=A, R={r}, R'=(],...,m}\R. 
Carry out elementary transformations to turn the r-th component of the x, 1 column 


to 1, and eliminate the other nonzeros in the column. By row and column exchanges, 
the resulting deficient-basis tableau is of the form 


Xn oe f |RAS 
1 Nr 1 
Nr 
—Nr —1| -1 


(19.16) 
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where Ne € R!*", Nr € R"—)*" are associated with the auxiliary feasible 
solution X,41; = 1, xy = 0 and objective value 1. It can be solved by the deficient- 
basis Algorithm 19.2.1. If the optimal value of the auxiliary problem vanishes, a 
feasible solution to the original problem is obtained (when x, ; leaves the basis); 
otherwise, there is no feasible solution to the original problem. 

In principle, any row having a nonzero right-hand entry can be taken as the first 
basic row. To maintain the sparsity of the deficient-basis tableau, however, one 
should select the row involving the minimum number of nonzeros since a dense 
basic row would cause a lot of fill-ins. Also, when the right-hand side is dense, the 
deficient-basis tableau would become dense, as consequently loses the value of the 
deficient-basis algorithm. 

Practically, the f column can be omitted from the initial tableau. The x41 
column can also be omitted because it is the same as the RHS column (except for 
the bottom row). In the following instance, the auxiliary objective row is placed at 
the bottom of the tableau and the original objective row is included as well so that 
it is convenient to enter Phase-II after the completion of Phase-lI. 


Example 19.4.1 
algorithm: 


Solve the following problem by the two-Phase deficient-basis 


min f = —3x1 + 2x2 — x3 + 3x5 + 2x7 — 6x8, 


st. —5x, +x2 +2x3 +2x4 +25 —1x7 —3xg = —15, 
—3x2 +4x3 4+3x4 —x5 —2x6 +2x7 +2xg = —11, 
—3x3 +5x4 —3x5 +x6 —3x7 +2xg = 2, 
+x3 +x5 —xg = 0, 
—X3 —X5 +2xg = 0, 
xj20, j=Hl,...,8 


Answer Phase-I: auxiliary initial tableau (the RHS column also represents the x9 
column) 


nal x2 X30 «X4 xs X6 X7 Xg ; RHS(x9) 
—5 1 2 1 1 -l -3 -15 
—3 1 3 -l -2 2 1 -l1 
=3 =3 1 -3 2 2, 
1 1 -1 
-1 -1 2 
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Iteration |: 

Multiply row | by —1/15, then add 11; —2; —1 times of row 2 to rows 2, 3, 7, 
respectively: 

The preceding auxiliary tableau is a feasible deficient-basis tableau: 


s=1, B={9}, R={l}, N={1,2,3,4,5,6,7,8}, R= {2,3,4,5}. 


Phase-I: call Algorithm 19.2.1. 


xX] x2 X3 X4 x5 X6 x7 Xg RHS(x9) 
1/3 1/15 2/15 1/15 1/15 1/15 1/5 1 
11/3* —56/15  —7/15 34/15 —26/15 —2 41/15 16/5 

—2/3 2/15 —41/15 TI/15S. —43/15 1 —47/15 8/5 

1 1 -1 

-1 -1 e) 

—3 2 -1 3 2 —6 
-1/3 1/15 2/15 1/15 1/15 1/15 —1/5 


Iteration 2: 


. min{—1/3, 1/15, 2/15, 1/15, 1/15, 1/15, —1/5} = —1/3 < 0, g=1. 

. Gg(R’) = (11/3, —2/3, 0, 0)? 40. 

. max{|11/3], | — 2/3|} = 11/3, i; = 2. 

. Multiply row 2 by 3/11, then add —1/3; 2/3; 3; 1/3 times of row 2 to rows 
1, 3, 6, 7, respectively. 

6.k=2, B= {9,1}, R= {1,2}, N = {2,3,4,5,6,7, 8}, R’ = (3,4, 5}. 


nA BW eRe 


xX] x2 X3 X4 X5 X6 x7 

afl 1/1 «= 3/11 1/11 2/11 —2/11 
1 56/55  —7/55 34/55 26/55 —-6/11 41/55 
—6/11* —31/11 61/11 —35/11 7/1 —29/11 

1 1 

-1 -1 
—58/55 —76/55 102/55 87/55 —18/11 = 233/55 
—3/11 1/11 3/11 —1/il —2/11 2/11 


RHS(x9) 


Iteration 3: 
1. min{—3/11, 1/11,3/11, —1/11, —2/11, 2/11, 1/11} = —3/11 <0, ¢q =2. 


3. Gq(R’) = (—6/11, 0,0)" 4 0. 
4. max{| — 6/11], 0,0} = 6/11, i, = 3. 


514 19 Deficient-Basis Method 


5. Multiply row 3 by —11/6, then add —3/11; 56/55; 58/55; 3/11 times of row 3 
to rows |, 2, 6, 7, respectively. 
6.k=3, B={9, 1,2}, R={1,2,3}, N = {3,4,5, 6,7, 8}, R’ = {4, 5}. 


xX] x2 x3 x4 X5 x6 x7 xg RHS(x9) 
—3/2 5/2* —3/2 1/2 —3/2 1 1 
1 T1/15 —146/15 82/15 —26/15 17/3, —16/5 
1 31/6 —61/6 35/6 —7/6 29/6 —4 
1 1 -1 
-1 -1 2 
61/15 —133/15 116/15 —43/15 28/3 = —38/5 
3/2 —5/2 3/2 —1/2 3/2 -1 


Iteration 4: 


. min{3/2, —5/2, 3/2, —1/2, 3/2, -1} = —5/2 <0, gq =4. 

. Gg(R’) = (0, 0)". 

. Ag(R) £ 0. 

. a =min{1/(5/2)} = 2/5, s = 1, is = 1. x9 leaves the basis. 

. Multiply row | by 2/5, then add 146/15; 61/6; 133/15; 5/2 times of row | to 
rows 2, 3, 6, 7, respectively. 


\O CON We 


10. B ={4, 1,2}, N = {3,5,6,7, 8, 9}. 

xX] x2 X3 x4 x5 X6 x7 Xg RHS(x9) 
—3/5 1 —3/5 1/5 —3/5 2/5 2/5 
1 —53/75 —28/75 16/75 = —13/75 52/75 292/75 
1 —14/15 —4/15 13/15 = —19/15 1/15 61/15 

1 1 -1 

-1 -1 2* 
—94/75 181/75 = —82/75 301/75 —304/75 266/75 


As the artificial x9 column has already become nonbasic, the preceding tableau 
is feasible. Delete the x9 column and the bottom row, then turn to Phase-II. 
Phase-II: call Algorithm 19.2.1. 
Iteration 5: 


. min{—94/75, 181/75, —82/75, 301/75, —304/75} = —304/75 < 0, gq = 8. 

. @g(R’) = (-1, 2)7 £0. 

. max{| — 1], |2|} = 2, i, = 5. 

. Multiply row 5 by 1/2, then add —2/5, —52/75, —1/15, 1, 304/75 times of row 
5 to rows 1, 2,3, 4, 6, respectively. 


an BWR 
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6. 


k=4, B= {4,1,2,8}, R= {1,2,3,5}, N = {3,5,6, 7}, R’ = {4}. 


Xp x2 X3 x4 x5 X6 x7 RHS 
—2/5 1 —2/5 1/5 —3/5 2/5 
i —9/25 —2/75 16/75 =—13/75 292/75 
1 —9/10 —7/30  =13/15 —19/15 61/15 
1/2* 1/2 
—1/2 —1/2 
—82/25 29/75 —82/75 301/75 266/75 


Iteration 6: 


nA B We 


. min{—82/25, 29/75, —82/75, 301/75} = —82/25 <0, gq =3. 

. Gg (R’) # 0. 

wdy =4. 

. Multiply row 4 by 2, then add 2/5, 9/25, 9/10, 1/2, 82/25 times of row 4 to 


rows 1, 2, 3, 5, 6, respectively. 


.k=5, B= {4,1,2,8,3}, R= {1,2,3,5,4}, N = {5,6, 7}, R’ = 9. 


xX] x2 X3 X4 x5 X6 X7 
1 1/5*  —3/5 
1 1/3 16/75  —13/75 
1 2/3 13/15 —19/15 
1 1 


11/3 —82/75 301/75 


Iteration 7: 


. min{11/3, —82/75, 301/75} = —82/75 <0, q =6. 

R' =@. 

. Ag(R) £ 0. 

. a = min{1/(20/87)} = 87/20, s =1, is = 1. 

. Multiply row 1 by 5, then add —16/75, —13/15, 82/75 times of row 1 to rows 
2, 3, 6, respectively. 

. B= {6,1,2,8,3}, N = {4,5, 7}. 
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Xp x2 x3 X4 x5 X6 X7 xg | RHS 
5 1 3 2, 
1 -16/15 1/3 T/15 52/15 
1 —13/3 Py hs: 4/3 7/3 
1 1 
1 
82/15 11/3 11/15 86/15 


Iteration 8: 


1. min{82/15, 11/3, 11/15} > 0. 
2. The optimal basic solution and optimal value: 


x* = (52/15, 7/3,0,0,0,2,0,0)',  f* =—86/15. 


19.5 Deficient-Basis Reduced Algorithm 


In this section, we incorporate the deficient basis into the reduced simplex algorithm 
to cope with large-scale and highly degenerate problems. 
Consider the reduced problem (16.1), i.e., 


min f, 


st. [A! ~ensil (7) =b, x>0. 


Assume that B, R are basic columns and rows, denoted by (19.1) (The row index 
m + 1 is the rth element of R); and N, R’ are nonbasic columns and rows, denoted 
by (19.2). Its initial tableau is then 


(19.17) 


xp xn | f |RHS 


Br Nr\—e,| br (19.18) 
Br Nr | bri 
where Bp is the deficient basis with the row index r € {1,..., k}, and —e, € Rk 
denotes the basic row part of dy41 = —é€m+1. Note that the f column is put alone 


as a special nonbasic column. _ 
Let f be given. Setting f = f and xy = 0, it can be determined from the first k 
rows of (19.18) that 


kp = By'br + f Baier, (19.19) 
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which determines the solution * to Ax = b with the objective value f. Then, if 
Br Bp er = 0, br — Br By'br=0, 


the tableau (19.18) can be transformed into the so-called deficient-basis reduced 
tableau below: 


XB XN | f Solution 


I Nri\w| xB (19.20) 
Ne 


If xg > 0, then (19.20) is a feasible tableau, and hence ready to carry out a rank- 
increasing or rank-remaining iteration (see Sect. 19.2). Note that the rank-increasing 
iteration is independent of both the f column and the xg column. 

Based on preceding discussions, integrating steps of Algorithms 16.1.1 
and 19.2.1 gives the following algorithm. 


Algorithm 19.5.1 (Deficient-Basis Reduced Algorithm: Tableau Form) Jnitial: 
feasible deficient-basis reduced tableau of form (19.20) with B, R, N, R', 1 < 


k<m-+1.xg, f. This algorithm solves the reduced problem (19.17). 


1. Stop if w = 0 (unbounded problem). 
2. Determine a and s such that a = —xj,/Ws = min{—x;,/w; | w;, < 0, t = 
1,..., ky}. 
. Ifa £0, update xg =Xgp+aw, f= f—a. 
. Determine column index q € argminjen Gi, j- 
Stop if ai,,q = 0 (optimality achieved). 
. Go to step 10 if R' = % or ag(R’) = 0. 
Determine row index p € arg maxjeR’ |Gpq|- 
. Convert ap q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
9. Setk =k +1, bring q from N to B and p from R' to R, and go to step 4. 
10. Convert aj, q to 1, and eliminate the other nonzeros in the column by 
elementary transformations. 
11. Update (B, N) by exchanging js and q. 
12. Go to step 1. 


CNAWAYW 


Note xg column is not changed in steps 8 and 10. 
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Example 19.5.1 Solve the following problem by Algorithm 19.5.1: 


min f = 3x2 + 2x4 + x5 — 4x6 — 2x7, 


S.t. xy —2x2 +3x4 +x5 —x6 —2x7 = 4, 
4x2 +x3 —5x4 +2x6 +3x7 = 12, 

6x2 —2x5 = 0, 

—5x2 —x7= 0, 

x2 —3x5 +4x7 = 0, 


Answer Initial:k = 2, B = {1,3}, R = {1,2}, and the feasible deficient-basis 
tableau is 


xX] x2 -X3 x4 x5 
1 —2 3 1 
4 1 —5 

6 —2 
—5 

1 —3 

3 2 1 


% = —4 < 0, Ge(R’) = (0,0, 0)", Take q = 6. Multiply row 6 by —1/4, then 
add 1, —2 times of row 6 to rows 1,2, respectively, obtaining the feasible deficient- 
basis reduced tableau below: 


xX] x2 X3 X4 X5 X6 X7 XB 
1 —11/4 5/2 3/4 8/2 4 
11/2 1 —4* 1/2 —4 12 
6 a1 
—5 1 
1 —3 4 
—3/4 =1j2. 14 1 1/2 


Call Algorithm 19.5.1:k = 3, B = {1,3,6}, R = {1,2,6}.N = 
{2, 4, 5, 7}, R’ = {3, 4, 5}. = 
Iteration 1: xg = (4,12,0)', f =0 


. w = (14, -1/2, 1/47 #0. 

. @ = min{—(12)/(—1/2)} = 24, s =2. 

. Add 24 time of f column to ¥g column; f = —24. 
. min{11/2, —4, 1/2, -44 = -4 <0, gq =4. 

. Gg(R’) = 0. 


DnRWNe 
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10. Multiply row 2 by —1/4, then add —5/2, 1/2 times of row 2 to rows 1,6, 
respectively. 
11. B={1,4,6}, R= {1, 2,6}, N = {2,3,5,7}, R’ = {3,4, 5}. 


x] x2 x30 X4 x5 x6 x7 f Xp 
1 1/l6 5/8 17/16 =4| i916. | 16 
11/8 -1/4 1 —1/8 1 | 1/8 

6 2) 

—5 -l1 

1 3 4* 

~23/16 —1/8 5/16 1 14 5/16 | 6 


Iteration 2: 


. w= (-1/16, 1/8, 5/16)" # 0. 

. a = min{—(10)/(—1/16)} = 160, s, is, js = 1. 

. Add 160 times of f column to xg column; f = —24 — 160 = —184. 

. min{11/16, 5/8, 17/16, —4}= —4 <0, q=7. 

. Ag(R’) = (0, -1,4740. 

. max{0, | — 1], |4]} =4, p=5. 

. Multiply row 5 by 1/4, then add 4, —1, 1, —1 times of row 5 to rows 1, 2, 4, 6, 
respectively. 

9.k=4, B=({1,6,4,7}, R= {1,2,6,5}, N = {2,3,5}, R’ = {3, 4}. 


CNIDNDHWNe 


xy x2 x3 x4 x5 X67 f XB 
1 27/16 5/8 —31/16 —1/16 
13/8 -1/4 1 5/8 1/8 | 20 
6 —1/2% 
—19/4 —3/4 
1/4 —3/4 1 
27/16 —1/8 7/16 1 5/16 | 56 


Iteration 3: 


4. min{27/16, 5/8, —31/16} = —31/16 <0, g =5. 

6. Gg(R’) = (—2, 3/4) £0. 

7. max{| — 1/2|, | — 3/4|} =3/4, p=4. 

8. Multiply row 4 by —4/3, then add 31/16, —5/8, 1/2, 3/4, —7/16 times of row 
4 to rows 1,2,3,5,6, respectively. 

9.k=5, B={1,6,4,7,5}, R= {1,2,4,5, 6}, N = {2,3}, R’ = {3}. 
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x1 x2 x3 x4 1X5 X67 f XB 
1 335/24 5/8 —1/16 
62 HAjfd 41 1/8 | 20 
55/6 
19/3 1 
5 1 
~107/24 —1/8 1 5/16 | 56 


Iteration 4: 


4. min{335/24, 5/8} > 0. 
5. The optimal basic solution and optimal value: 


x* = (0,0,0,20,0,56,0)', = f* = —184. 


It is obvious that the revised version of Algorithm 19.5.1 has features of both 
deficient-basis and reduced simplex methods. Assume that Br is the deficient basis, 
then the revised deficient-basis reduced tableau is 


XB XN f Solution 
1 Br Nr —Brier| XB (19.21) 
Ne — Br Bp Nr 


Based on equivalence between (19.20 and (19.21), it is straightforward to obtain: 


Algorithm 19.5.2 (Deficient-Basis Reduced Algorithm) Initial: 1 < k <m+ 
1, B, R, BS f;w= —Bz' er, XB= Br'br — fw > 0. This algorithm solves 
the reduced problem. 


1. Stop if w = 0 (unbounded problem). 


2. Determine a and s such thata = —xj,/Ws = min{—xj;,/wW; | w;, < 0, t = 
1,..., ky}. 
3. Ifa £0, update Xp = Xp + aw, f = f—a. 
4. Compute oy = NIB, ex: 
5. Determine column index q € arg minjen Gj. 
6. Stop if oq = 0 (optimality achieved). 
7. Compute Gg(R) = By'aq(R). 
8. Go to step 13 if R’ = Y or dg(R’) = dag(R’) — Bridg(R) = 0. 
9. Determine p € arg maxjeR’ | (di, q|- 
10. Update By’ by (19.10). 
Il. Setk=k+1, Xg =0, Gk,n41 = 0, and bring q from N to B and p from R’ 
to R. 
12. Go to step 4. 
13. Compute & = Gj, n+1/4is, q- 
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14, Ifa £0, set ws = a, and update w = w — ad, (R). 
15. Update By’ by (19.11). 

16. Update (B, N) by exchanging js and q. 

17. Go to step 1. 


Note Phase-I procedure is usually needed to get the preceding algorithms started. 
To this end, construct an auxiliary problem by introducing artificial variable f with 
added column 5 in the coefficient matrix; then starting from (x = 0, f = 1), 
minimize objective f to zero. 


Example 19.5.2 Solve the following problem by the preceding algorithm: 


min f = —3x; +x4 + 4x5 — 2x6 + 247, 


s.t. —3x1 +x3 —x4 —2x5 +4x6 —2x7 = 9, 
+x2 +4x4 +3x5 —3x6 —S5x7 = 2, 

6x1 —6x4 —2x5 = 0, 

+4x7 = 0, 

xX] —x4 —x7 = 0, 


xp 0, FSU, 25257. 


Answer The reduced problem is associated with the following tableau: 


xX] x2 x3 X4 xs X6 x7 f | RHS 
—3 1 -l -2 4 -2 9 
1 4 3 =3 =5 2 
6 -6 -—2 
4 
1 -1 -1 
—3 1 4 -2 2] -1 


Initial: The preceding is a feasible deficient-basis tableau. A feasible reduced 
(deficient) basis is yielded by transforming the bottom row to a basic row and 
xe column to a basic column, ie., kK = 3, B = {3,2,6}, R = {1,2, 6}, 
N ={1,4,5, 7}, R’ = {3, 4, 5}. 


1 4 i 2 
Br=| 1-3]. By =| 1-3). 
—2 —1/2 
—2 9 


Gin41 = —Bzle3 =| 3/2]. f=0, p= Bplbr=| 2] >0. 
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Iteration 1: 
1. w = (—2, 3/2, 1/2)" # 0. 
2. a = min{—9/(—2)} = 9/2, s =1. 
3. Xp = (9, 2,0)? + (9/2)(—2, 3/2, 1/2)" = (0, 35/4,9/4)", f = —9/2. 
T —9 


HB ei 9 1 2 1 ; 
sov=( 4 3) ( 17] (:)- : 
=3,-h 4: 2 —1/2 0 s 
5. min{—9, 1,6,2}=—-9 <0, g=1. 
1 2\ /-3 -~9 
7. dg(R) = ( 1] ( 7 = (22) 
iio} W83 3/2 
6 000 6 6 
8. dg(R’) = (") = (200) (=) = (") #0. 
1 000/ \-2 1 
9. max{|6], 0, |1|} = 6, p = 3. 
10. t = (6— (0, 0, 0)(—9, 9/2, 3/2)')-! = 1/6. 


v = —(1/6)(—9, 9/2, 3/2)? = (3/2, —3/4, -1/4)". 


1 2 
d' = —(1/6)(0, 0, 0) ( 1 2) = (0, 0, 0). 


-1/2 
1 ps 3/2 1 2 
U=| 1-3/2]-]| -3/4](000)=] 1-3/2]. 
—1/2 —1/4 Ea) 
1 2 3/2 
g-1_| 1-3/2 -3/4 
R —1/2 -1/4 
1/6 
Hl. k=4, B= (3,2,6,1), R=(1,2,6,3), N=(4,5,7}, 
R’ = {4,5}, By’ = Be. X1 = 0, Ganzi = 0. 
Iteration 2 
BN 71 2° 3)2\ "74 “a 
4 3-5 1 —3/2 -3/4 0 
4. on = = 3 
i Ae 1/2 -1/4 0 ; 
O22 1/6 0 
5. min{—8, 3,2} = —13/2 <0, gq =4. 
1 2 3/2\ /-1 -8 
7. Gg(R) = 1 —3/2 -3/4 4} _ 
—1/2 -1/4 1 


1/6) \-6 = 
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—8 
seas 0 0000 7 0 
. a= (_9)-(go01) 1 =(5). 
-1 
13. @ = —2/-8=1/4. 
14. w = (2, 3/2, 1/2, 0)" — (1/4)(-8, 7, 1, -1)? = (0, -1/4, 1/4, 1/4)", 
wy, = 1/4. 
1/8 \fl 2 3/ —1/8-1/4-3/1 
15. Bria| 7/8! |) -3/2-3/4]_] 7/81 1/4 9/16 
PSR 1/81 || —1/2-1/4 1/8—1/4-1/16] 
—1/8 ] 1/ —1/8—1/4-1/4 


16. B={4,2,6, 1}, N= 3,5,7}; Bp! = B.. 
Iteration 3: 

1. Gng1 = (1/4, -1/4, 1/4, 1/47 # 0. 

2. wa = min{—(35/4)/(—1/4)} = 35, s = 2. 


3. ¥p = (0, 35/4, 9/4, 0)" + 35(1/4, —1/4, 1/4, 1/4)" = (35/4, 0, 11, 35/4)", 
f =—-9/2 — 35 = —79/2. 


L2BV (H1/e? Sve sey <0 a 
3-5 7/81 1/4 9/16) [1 / 
= ={ 9s). 
4 2 8 -1/4-1/16] [0] \ ae), 
=) 1/8 —1/4-1/48/ \o 
5. min{7/8, 9/8, 25/4} = —25/4 <0, q =7. 
=f Sa Sa ey (= -1/4 
7 acrya| 7/81 W4 of] | -5] _| 25/4], 
1/8 —1/4 -1/16 2 —3/4 
1/8 —1/4—1/48 0 -1/4 
1/4 
— on _( 4\_( 0000\] -25/4] (4 
. ae’) =(_1) Coa) ~3/4 ats): 
-1/4 


9. max{|4],|—1]}=4, p=4. 
10. r= (4-0) !=1/4, 
v = —(1/4)(—1/4, —25/4, —3/4, —1/4)™ = (1/16, 25/16, 3/16, 1/16)". 
d™ = —(1/4)(0,0,0,0)Bz' = (0, 0,0, 0). 
U = Bz! — (1/16, 25/16, 3/16, 1/16)7(0, 0, 0, 0) = BR!. 
-1/8 -1/4-3/16 1/16 
7/81 1/4 9/16 25/16 
Be’ =| 1/8 -1/4-1/16 3/16 }. 
1/8 =1/4'=1/48 (1/16 
1/4 
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11. k=5, B= {4,2,6,1,7}, R={1,2,6,3,4}, N = {3,5}, R’ = {5}. 


R 
Iteration 4: 
T aE 

1 —2 1/8 1/4 —3/16 1/16 0 
3 7/81 1/4 9/16 25/16 1 7/8 
4. on= 4 1/8 -—1/4-1/16 3/16 0 =() 

2 1/8 1/4 -1/48 1/16 0 

1/4 0 


5. min{7/8, 9/8} > 0. 
6. The optimal basic solution and optimal value: 


x* = (35/4, 0,0, 35/4,0,11,0)7,  f* =—79/2. 


Note In general, a Phase-I procedure is needed to get the preceding algorithms 
started. To this end, an auxiliary problem can be formed by introducing artificial 
variable f with added column b in the coefficient matrix. It is clear thatx = 0, f = 
1 is feasible to the problem. Thus, what we to do next is to minimize objective f to 
reach zero. 


19.5.1 Phase-I: Most-Obtuse-Angle Rule 


As was seen in Sect. 19.4.3, a feasible deficient-basis tableau can be created by 
solving the auxiliary program (19.15) using the deficient-basis algorithm. Since the 
program is itself in reduced form; it seem to be attractive to use the deficient-basis 
reduced algorithm instead. 

To begin with, put the initial tableau, say (19.15), into the following form: 


x | f | RHS 
Albl| 6 


Take x = 0, f = 1as the initial solution. Assume that the current deficient-basis 
tableau is 


XB XN f RHS 
I Nr br br 
Ne 
where B = {ji,..., je}, R= {ii,...,,4 }. It represents the equalities 


xp + Nrxw =—brf +br, Nrxn = 0. 
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If the set 
S={t | b, <0,t=1,...,k} 


is empty, then f can be decreased from 1 until reaching 0, giving a feasible solution 
to the original problem; i.e., 


Xp = br >0, Xn = 0. 
If S is nonempty, the deficient-basis reduced algorithm determines a row index 
s €argmin —xj,/Di,- 
teS , 
In our case, there is usually a tie in selection of s, since x;, = 0, Vt € S. As the 
associated stepsize vanishes, no solution updating should happen. For such a highly 
degenerate case, we use the following rule instead: 
s €argmin bi,. 
tes 
which is not only stable but also advantageous in the sense of the most-obtuse-angle 


heuristic. 
Therefore, the preceding tableau is simplified to 


xt xt | RHS 
I Nr br (19.22) 
Nr 


The subsequent steps are similar to Algorithm 19.5.1, as is formulated as follows. 


Algorithm 19.5.1 (Deficient-Basis Phase-I: The Most-Obtuse-Angle Rule) Ini- 
tial: deficient-basis tableau of form (19.22) with B, R, N, R’, 1 < k < m. This 
algorithm finds a feasible deficient-basis tableau. 

The same as Algorithm 19.5.1 except for its steps 1-5 replaced by 


. Determine s € arg min{b;, | t=1,...,k}. 
. Stop if b;, > 0 (feasibility achieved). 

. Do nothing. 

. Determine q € argminjen dij,, ;. 

. Stop if aj,., = 0 Gnfeasible problem). 


nA WN Ee 


To get the preceding Algorithm started, the procedure described in Sect. 19.4.1 
can be used to gain an initial deficient basis. For matching the Algorithm, it seems 
to be a good choice to utilize the following. 
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Algorithm 19.5.2 (Producing of Initial Basis) Initial: (A = A |b = b), B = 
0, R=0,N=A, R’={I,...,m}, k = 0. This algorithm finds a deficient-basis 
tableau. 


. Determine p € argmaxjer’ |bj|. 

. Stop if |b p| = 0 (deficient-basis attained). 

. Stopif J ={j | apjbp > 0, j € N} = (infeasible problem). 

. Determine q € argmaxjey |p ||. 

. Convert dp q to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 

6. Setk =k + 1, bring q from N to B and p from R’ to R. 

7. Go to step lif R’ #9. 

8. Stop (full basis attained). 


nABWN 


Example 19.5.3 Solve the following problem : 


min f = 7x; + 2x2 + 3x3 — 2x4 + 9x7 — 2x8 


s.t. —5x, — x2 —5x34+2x4+ x5 + x6 —3x7-—3x8 =-9, 
2x, +2x2 —2x34+ x4+ 3x5 — xo + x7 — 2x8 = -9, 

5x1 — 3x2 — 3x44 5x5 — 3x6 +2x7+ xg = 2, 

3x2 + x4 + 3x7 = 0, 

—4x7 = 0, 


Answer(1) Call Algorithm 19.5.2 to determine an initial basis. The equality 
constraints can be represented by the following tableau: 


Pal x2 x3 X4 x5 X6 Xx7 Xg RHS 
-5 -1 -5 2 1 1 -3 -3 -9 
2 2 2 1 3 -l 1 -2 —9 
5 —3 —3 5 -3 2 1 2 
3 1 3 
—4 


corresponding to B=, N ={1,...,8}, R=@, R’={l,...,5}, k=0. 
Iteration 1: 


1. max {| — 9], | — 9], |2|} =9, p=1. 

4. max {| — 5],]— 1],]—S],1—31,1-3)}}=5, q=1. 

5. Multiply row 1 by —1/5, then add —2,—5 times of row 1 to rows 2,3, 
respectively. 

6.k=1, B={l}, N= ({2,...,8}, R={l}, RB’ = {2,..., 5}. 
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=4 +5 =I 6 —2 = 
3 1 3 
—4 


Iteration 2: 


1. max {| — 63/5], | — 7|} = 63/5, p =2. 

4. max {| — 4], | — 3/5], | — 1/5], | — 16/5|} = 4, ¢ = 3. 

5. Multiply row 2 by —1/4, then add —1,5 times of row 2 to rows | and 3, 
respectively. 

6.k=2, B= {1,3}, N = {2,4,...,8}, R= {1,2}, R’ = (3,4, 5}. 


xX] x2 X3 X4 X5 X6 x7 
1 3/5 1/20 13/20 —7/20 11/20 
—2/5 1 —9/20 —17/20 3/20 1/20 

-~6 —13/4 1/4 -5/4 3/4 

3 1 3 

—4 


Iteration 3: 


1. max {|35/4|}, p = 3. 

4. max {|7/4], |2|} = 2, gq =8. 

5. Multiply row 3 by 1/2, then add 1/5, —4/5 times of row 3 to rows | and 2, 
respectively. 

6.k=3, B= {1,3,8}, N = {2,4,5,6, 7}, R= {1, 2,3}, R’ = {4, 5}. 


xX] x2 x3 X4 X5 X6 
1 11/40 33/40 ~~ —19/40 
2 1 17/20 = —31/20 13/20 
=.) 19/8 7/8  —5/8 
3 1 


A deficient basis attained 


(2) Call Algorithm 19.5.1 to achieve feasibility. 
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Iteration 4: 


1. min {—19/40, —7/20, 35/8} = —19/40, s = 1. 

3. min {—11/40, —19/40} = —19/40, q = 6. 

6. dg(R’) = 0. 

0. Multiply row | by —40/19, then add —13/20, 5/8 times of row | to rows 2,3, 
respectively. 

ll. k=3, B= {6,3,8}, N = {1,2,4,5,7}, R = {1,2, 3}, R’ = {4,5}. 


x1 x. x3 x4 x5 X6  X7 xg | RHS 
—40/19 11/19 —33/19 1 -1l 1 
26/19 2, 1 9/19 —8/19 1 -1 
—25/19 —-3 —24/19 —4/19 -1 1 5 
3 1 3 
—4 


Iteration 5: 


1. min {1, —1,5} =—1, s =2. 

3. min {—8/19}, g =5. 

6. dg(R’) = 0. 
10. Multiply row 2 by —19/8, then add 33/19, 4/19 times of row 2 to rows | and 


3, respectively. 
ll. k=3, B= {6,5, 8}, N = {1,2,3,4, 7}, R= {1,2,3}, R’ = {4,5}, 


Feasibility is attained: xg = (41/8, 19/8, 11/2)", xy = 0. Add the objective 
row at the bottom and f = f column, and add 2 times row 3 to the bottom line to 
transform it to a feasible deficient-basis tableau: 


Xx] x2 x3 X4 Re ke X7 = X8 f RHS 
31/4 33/4 33/8 11/8 1 —41/8 41/8 
13/4 19/4 19/8 —9/8 1 —19/8 19/8 
—2 —4 -1/2 —3/2 —3/2 1 11/2 
3 1 3 
—4 
3 —6 2: —5 6 -1 11 


with solution 
x = (0,0, 0, 0, 19/8, 41/8, 0, 11/2, 0)', f =-ll. (19.23) 


(3) Convert the preceding to a reduced tableau. 
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Iteration 6: 
J = {2,4}, min{—6, —5} = —6, q = 2, ag(R’) £0. 

Multiply row 4 by 1/3, then add 33/4, 19/4, 4, 6 times of row 6 to rows 1, 2, 3, 6, 
respectively. 

k=4, B= {6,5,8,2}, N = {1,3,4,7}, R= {1,2,3,4}, R’ = {5}. 


XxX] x2 X3 X4 x5 X6 x7 
—31/4 —33/8 11/8 1 25/8 
—13/4 19/8 11/24 1 19/8 

a) =i 176 5/2 

0 1 0 1/3 1 

0 0 -4 

3 2 a3 12 


Iteration 7: 
Put xg column in place of RHS column (by zeroing the value 11 at the south-east 
comer). 

J = {4}, q=4, agR’ =0. 

Multiply row 6 by —1/3, then add —11/8, —11/24, 1/6, —1/3 times of row 6 to 
rows 1, 2, 3, 4, respectively. 


x] x2 x3 X4 X5 X6 x7 XB 
—51/8 —77/24 1 69/8 41/8 
—67/24 —149/72 1 101/24 19/8 
—13/6 —11/18 11/6 11/2 

1/3 1 2/9 7/3 

0 0 -4 

-1 —2/3 1 —4 


k=5, B= {6,5,8,2,4}, N = {1,3,7}, R = {1, 2, 3,4, 6}, R’ = {5}. 
(4) Call Algorithm 19.5.1: 
Iteration 7: 


. w = (—11/24, —11/72, 1/18, —1/9, 1/3)" # 0. 
.a=0,5s=4. 

. min{1/3, 2/9, 7/3} = 0. 

. Optimality achieved by solution (19.23). 


APN Re 
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Chapter 20 ®) 
Dual Deficient-Basis Method Cheek for 


This chapter attacks the standard LP problem from the dual side using the deficient 
basis. To achieve optimality, the method presented in the previous chapter seeks 
dual feasibility while maintaining primal feasibility. Seeking the primal feasibility 
while maintaining dual feasibility, in this chapter, we derive the dual deficient-basis 
algorithm in tableau form and its revised version. Combining the deficient-basis and 
D-reduced algorithms, we also develop the dual deficient-basis D-reduced algorithm 
in tableau form and its revised version and the deficient-basis D-reduced gradient 
algorithm in tableau form and its revised version. 


20.1 Dual Deficient-Basis Algorithm: Tableau Form 


Assume that the deficient-basis tableau (19.4) is dual feasible, satisfying z; > 0. If 
b > 0, optimality is already achieved; else, determine row index i, such that 


is € argmin b;, <0. (20.1) 
i;ER 


This will let the basic infeasible variable xj, leave the basis, becoming a feasible 
variable. 


Lemma 20.1.1 Assume that zy > 0 and bi, < 0. If the index set 
J={j EN |G, j < 0} 


is empty, then problem is infeasible. 


Proof It is seen that since bi, < 0, the isth row of the tableau actually gives an 
upper-hill direction, with respect to the dual objective function. If the entries of 
this row are all nonnegative, then the stepsize taken along this direction is allowed 
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arbitrarily great while maintaining dual feasibility; that is to say that the dual 
problem is unbounded, and hence there is no feasible solution to the standard LP 
problem. Oo 


Now assume that J ¢ 4. The maximum possible stepsize that can be taken is 
p= —Zq/di, q = min{—z;/di, j | di, jx< 0, jE J} = 0, (20.2) 


where column index q corresponds to a dual constraint, broking increasing of the 
dual objective value. Adding 6 > 0 times of the i;th row to the bottom (objective) 
row updates the dual solution. Consequently, 7, vanishes, and the southeast entry of 
the tableau becomes 


—f=—-f+ bi, <—-f, 


which means that the objective value f does not decrease, and it strictly increases 
under the assumption of dual nondegeneracy (8 > 0). 
There will be the following two cases to be handled differently: 


(i) Rank-remaining iteration: R’ = % or G(R’) = 0. 
Convert aj,,, to 1, and eliminate the other nonzeros in the column by ele- 
mentary transformations. Consequently, the j,-indexed basic column becomes 
nonbasic, and the g-indexed nonbasic column becomes basic, while the basic 
rows remain basic. So, the rank of the basis remains. This turns bi, to positive 
but does not touch nonbasic rows at all (including entries in the right-hand side). 

(ii) Rank-increasing iteration: R’ 4 @ and ag(R’) # 0. 
Determine row index p € arg maxjepR’ |@jq|. Convert dp, to 1, and eliminate 
the other nonzeros in the column by elementary transformations. Then, while 
all basic columns and rows remain basic, the g-indexed nonbasic column and 
p-indexed nonbasic row become basic. Hence the rank of the basis increases by 
1. Since the right-hand side remains unchanged, the same row index i, will be 
selected in the next iteration, until encountering a rank-remaining iteration that 
makes b;, positive. 


The associated steps can be summarized into the following algorithm. 


Algorithm 20.1.1 (Dual Deficient-Basis Algorithm: Tableau Form) Initial: a 
dual feasible deficient-basis tableau of form (19.4) with B, N, R, R’, 1<k<™m. 
This algorithm solves the standard LP problem. 


Determine s such that i; € arg minj,cr bj,. 

Stop if b;, > 0. 

Stop if J = {j € N | aj, ; < O}. 

Determine column index g and stepsize B by (20.2). 

If 6 ~ 0, add £ times of the i, th row to the bottom row. 
Go to step 10 if R’ = Y or ag (R’) = 0. 

Determine row index p € arg maxjeR’ |G ql. 


SOS ve 
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8. Convert dp, to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
9. Setk =k + 1, bring q from N to B and p from R’ to R, and go to step 3. 
10. Convert dj, g to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
11. Update (B, N) by exchanging j; and q. 
12. Go to step 1. 


Theorem 20.1.1 Under the dual nondegenerate assumption, Algorithm 20.1.1 
terminates either at 


(i) Step 2, reaching a basic optimal solution, or at 
(ii) Step 3, detecting the infeasibility of the problem 


Proof The proof of finiteness of the algorithm is similar to that in the conventional 
simplex context. The meanings of its exits come from Lemmas 19.1.1 and 20.1.1, 
as well as discussions preceding the algorithm. Oo 


Example 20.1.1 Solve the following problem by Algorithm 20.1.1: 


min f = 4x3 + x4 + 3x5 +.x%6 + 3x7 + 2x8, 


st. xX] +2x3 —x4 +x5 —x6 —3x7 —3xg = —10, 
+x2 +23 +2x4 —x5 +x6 —2x7 +xg= —7, 

3x3 X4 —3x5 X6 +2xg = 0, 

+x3 +x4 +2x5 +3Xx6 —xg= 0, 

—x3 +X4 —X5 —2X6 +2xg = 0, 


xj 20, jHl,...,8. 


Answer s = 2, B = {1,2}, R= {1,2}. N = (3,4,5, 6,7, 8}, R’ = (3,4, 5}. The 
initial feasible dual deficient-basis tableau is 


XxX, x2 x3 x4 x5 X6 X7 xg | RHS 
1 2 -1l 1 1 3 3 10 
1 1 2 -il 1 -2 1 -—7 
3 1 3 1 2% 
1 1 1 3 -1 
-1 1 -l -2 2: 
4 1 3 1 3 2 


Iteration 1: 


1. min{f—10, —7} = -10 <0, s=1, is = 1. 

3. J = {4, 6,7, 8} 4D. 

4. B = min{—1/(—1), —1/(-1), —3/(—3), —2/(—3)} = 2/3, q = 8. 
5. Add 2/3 times of the first row to the bottom row. 
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~ 


. Gq (R’) = (2,-1, 2) £0. 
- max{|2|,|— 1], |2|}=2, p =3. 
. Multiply row 3 by 1/2, and then add 3, —1, 1, and —2 times of row 3 to rows 1, 


2, 4, and 5, respectively. 


_k=3, B= {1,2,8}, r={1, 2,3}, N = {3,4,5, 6,7}, R’ = {4, 5}. 
Bal x2 x3 XA x5 X6 X7 = X8 RHS 
1 5/2 5/2 142, 5/2. 3 —10 
1 5/2 5/2 1/2 3/2 —2 7] 
3/2 1/2 3/2 1/2 1 
-1/2 1/2 -1/2 5/2 

2 2* 2 -1 

2/3 16/3 1/3 11/3 1/3 1 —20/3 


Iteration 2: 


aOerANNN 


. J = {3,4,5,6, 7} 4G. 
- B=min{—(16/3)/(—5/2), —1/3)/(—5/2), —(1/3)/(—7/2), — 1/3) /(-5/2), 


—1/—3} =2/15, q=4. 


. Add 2/15 times of the first row to the bottom row. 

. Gig(R’) = (-1/2, 2)? £0. 

. max{|1/2], |2|} = 2, p =5. 

. Multiply row 5 by 1/2, and then add 5/2, —5/2, 1/2, and —1/2 times of row 5 


to rows 1, 2, 3, and 4, respectively. 


9.k=4, B={1,2,8,4}, R= {1,2,3,5}, N = (3,5,6,7}, R’ = {4}. 
Eat x2 x3 X4 X5 X6 x7 xg | RHS 
1 -l1 -15/4 -3 —10 
1 —2 11/4 -—2 -—7 
= = -3/4 1 
-l1 —1 11/4* 
11 1 -1/2 
4/5 5 16/5 3/5 —8 
Iteration 3: 
. J ={5,6, 7} 4G. 


3 
4 
6. 
7 
8 


B = min{—(16/5)/(—5/2), 0/(—15/4), —(3/5)/(—3)} = 0, g = 6. 
Gg(R’) = (11/4) £0. 


. max{{11/4]} = 11/4, p =4. 
. Multiply row 4 by 4/11, and then add 15/4, —11/4, 3/4, and 1/2 times of row 4 


to rows |, 2, 3, and 5 respectively. 
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9.k=5, B={1,2,8,4, 6}, R=({1,2,3,5,4}, N =(3,5,7}, R =@. 


bal x2 X3 X4 x5 X6 X7 xg | RHS 
1 —15/11 —26/11 —3* —10 
1 1 —1 —2 —7 
-—14/11 —14/11 1 
—4/11 —4/11 1 
9/11 1 9/11 
4/5 5 16/5 3/5 -8 


Iteration 4: 


3. J= (3,5, 7} 4 VY. 

4. 6B = min{—5/(—15/11), —(16/5)/(—26/11), —G/5)/ — 3} = 1/5, gq =7. 

5. Add 1/5 times of the first row to the bottom row. 

6. R=. 

10. Multiply row 1 by —1/3, and then add 2 and —3/5 times of row 1 to rows 2 
and 6, respectively. 

11. B= {7, 2, 8,4, 6}, N = {1, 3, 5}. 


xX] x2 x3 x4 x5 X6 7 xg | RHS 
-1/3 5/11 26/33 1 10/3 
—2/3* 1 21/11 19/33 —1/3 

—14/11 —14/11 1 
—4/11 —4/11 1 
9/11 1 9/11 
1 52/11 30/11 —10 


Iteration 5: 


1. min{10/3, —1/3} = —1/3 <0, s=2, i; =2. 

3. J={IFoB. 

4. 6 = min{—1/(—2/3)} = 3/2, g=1. 

5. Add 3/2 times of the second row to the bottom row. 

6. R =O. 
10. Multiply row 2 by —3/2, and then add 1/3 and —1 times of row 2 to rows 1 


and 6, respectively. 
11. B= {7,1,8,4, 6}, NM = {2, 3, 5}. 


Iteration 6: 


1. min{7/2, 1/2} > 0. 
2. The optimal basic solution and optimal value are 


x* = (1/2,0,0,0,0,0,7/2,0)!,  f* = 21/2. 
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x1 x2 X3 X4 x5 Xe x7 XB RHS 
—1/2 —1/2 1/2 1 7/2 
1 —3/2  -—63/22 —19/22 1/2 
—14/11 —14/11 1 
—4/11 —4/11 1 
9/11 1 9/11 
3/2 167/22 79/22 —21/2 


20.2 Dual Deficient-Basis Algorithm 


Since all quantities involved in Algorithm 20.1.1 can be found in Table 19.1, it is 
straightforward to obtain its revised version. 

Algorithm 20.2.1 [Dual Deficient-Basis Algorithm] Initial: B, N, R, R’, 1 < 
k <m. B's zn = cn — N} By’ cp > 0; Zp = 0; ¥ = By'bp. This algorithm 
solves the standard LP problem. 


1. Determine s such that xj, = min{x;,|f—=1,..., k}. 

2. Stop with f = cle B if x;, = 0 (optimality achieved). 

3. Compute oy = NJ Bptes. 

4. Stop if J = {j € N | o; < 0} = @ (infeasible problem). 

5. Determine 6 and column index g such that B = —Z,/og = minjey —Z;/0j;. 
6. If 6B #0, update Zzy = zy + Bon, 2), = Zj, + B. 

7. Compute dg(R) = B'ag(R). 

8. Go to step 12 if R’ = J or dg (R’) = ag(R') — Brrdg(R) = 0. 

9. Determine row index p € arg maxjeR’ |Giq|. 
10. Update Bz’ by (19.10). 
11. Updatek =k+1, x, = 0, bring q from N to B and p from R’ to R, and go to 

step 3. 

12. Compute a = Xj, /dj,.q- 
13. Ifa £0, set x, =a, and update xg = xp — aag(R). 
14. Update Bp’ by (19.11). 
15. Update (B, N) by exchanging j, and q. 
16. Go to step 1. 


The preceding algorithm needs a dual feasible tableau (solution) to get itself 
started. The (conventional) dual simplex Phase-I methods (Chap. 15) are applicable 
to this purpose, almost without any modification. 

Originally, the deficient basis is not defined as a square but a rectangular matrix, 
whose column space includes the right-hand side (Pan 2005). As the associated 
algorithm still reflects the behavior of Algorithm 20.2.1, we quote the computational 
results in the article as follows: 

The computational experiments were carried out on a Pentium III 550E PC 
with Windows 98 operating system, 256MB inner storage, and about 16 decimal 
precision. Visual Fortran 5.0 compiler was used. 
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Table 20.1 


Problem Iteration counts | Time | % Degen 
Small (20) 1.02 1.56 | 1.08 
Medium (15) 1.10 1.06 | 0.89 
Large (15) 1.44 1.25 | 1.07 
Average (50) ‘| 1.37 1.24 | 1.02 
Kennington (8) | 5.68 2.83 | 0.30 
BPMPD (5) _| 3.08 1.41 | 0.92 
Netlib (2) 0.94 1.01 | 1.80 
Average (15) 4.58 2.14 | 0.47 


The following two codes were tested and compared: 


1. MINOS 5.51 
2. RDPPA 1.10 


The second code was developed by modifying MINOS 5.5.1 with minor changes. 
The test set of 65 standard LP problems was the same as that used in Sect. 19.3.1, 
classified to the same sets. 

Table 20.1 lists iterations and time ratios (MINOS 5.51/RDPPA 1.10), in which 
the last column gives degenerate iterations percentage ratios. 

It is seen from the preceding table that the new code significantly outperformed 
MINOS 5.51 overall. For the first group of problems, the time ratio is 1.24, and 
for the second group, the time ratio reaches 2.14. Thus, the superiority margin is 
higher with large-scale sparse problems. Comparing these results with those listed 
in Sect. 19.3.1, we conclude that the dual deficient-basis algorithm appears to be 
better than its primal counterpart. 

From the last column of the preceding table, it is seen that the degenerate iteration 
ratio for the first group is 1.02, very close to 1, whereas that for the second group 
of larger problems is significantly less than 1. Overall, the proportion of degenerate 
iterations is high with the new algorithm. Therefore, it is not true that degeneracy 
itself degrades the algorithm’s efficiency. 


Example 20.2.1 Solve the following problem by Algorithm 20.2.1: 


min f = 4x3 + x4 + 3x5 +x6 + 5x7 + 2x8, 


s.t. xX] +2x3 —x4 +x5 —x6 —3x7 —3xg = —9, 
+x2 +x3 +2x4 —Xx5 +xX6 —2x7 +xg = —4, 

—3x3 —3x5 +2x3g = 0, 

+X3 +x5 —xg= QO, 

—X3 x5 +2xg = 0, 


xj20, j=l,...,8. 


Answer Initial: s— 2, B = {1,2}, R = {1,2}. N = (3,4,5,6,7,8}, R’ = 
(3.4.5) Be = J gy = 4, 1,3,1,5,2)" Ze = 0, Fe = (0,4). 
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Ite 


ration 1: 


_ min{—9, -4} = —9 <0, s=1. 
PS a1 ee oy 
J ={4,6,7, 83} £9. 
. B=min{—1/(—1), —1/(-1), —5/(-3), —2/(—3)} = 2/3, q = 8. 
Bw SA 13-150)! FONG. —1, b= 1,-3.3)! 116/31 11/3;1 
/3,3, 0)!. 

N =(3,4,5,6, 7,8}, 71 =0+2/3 = 2/3. 


7. Gq(R) = (3, IF. 


11 


Ite 


. Gg (R’) = (2,-1,2)? £0. 
. max{|2|,|—1|, |2)} = 2, p =3. 
. p=3, t= (2— (0, 0)(-3, I) )-! = 1/2. 
v = —(1/2)(—3, 1)? = (3/2, -1/2)', dl = (0,0). 


v=(}-Cayon=(}) 


1 3/2 
peat = 
Bet =| 1-1/2). 
1/2 
_k=3, B={1, 2,8}, R= {1, 2,3}, xg =0, N = {3,4,5,6, 7}, R’ = {4, 5}. 


ration 2: 
T 


22h 13 1 
_s=l,oy=| 1 2-1 1-2 TM en sy eae er eens eee) 
23° 3 0 3/2 


. J=(3,4,5,6, FAD. 
. B = min{—(16/3)/(—5/2), —d/3)/(-D), —1/3)/(-7/2), —d/3)/(-D, 
—(3)/(-3)} = 1/3, q=4. 
. Ey = (16/3, 1/3, 11/3, 1/3, 3)! + (1/3)(—5/2, —1, -7/2, -1, -3)7 
= (9/2, 0, 5/2,0,2)!, NW = (3,4, 5,6, 7}, 71 =2/3+1/3=1. 


i. 3\./=4 4 

.ag(R)=[{ 1-1/2 re ee a 
1/2 0 0 
=] 

~ py _(0\_ (00-1 0 
ane (3)-(3)(2}=(0) 
i SDV (1) 9: 
9: kp 9 0) 91.9 OF 2 0.29.0) 


= 1 3/2 Sr, 2 
ai-(n )( r-in)= (7 va) 
1 1/2 1/2 


. B= {4,2, 8}, N = {1, 3,5, 6, 7}. 
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Iteration 3: 


1. 


1 2 1-1-3\' /2 
on = La. £3  (20.25/2,=137> =1, 8)". 
3-3 5/2 


.S=(3,5,6, £90. 
. B = min{—(9/2)/(—5/2), —(5/2)/(—13/2), 0/(—1), —2/(—8)} = 0, q =6. 


a1 =3/2\. fi 1 5 
.G(R)={ 21 5/2 1)=[-1], a(R) = i 
1/2 0 0 


. Gig(R’) = 0. 
(2) (1) S99. 
ke = 0,.—22:0)' — 000,14, 0) =] G13;0.0)5, = 


ee = °=3/2 
| = | )( ya a) =(-2-1 a 
1 1/2 1/2 
B 


min{9, —22,0} = —22 <0, s=2. 


= (4,6, 8}, N = (1, 2, 3,5, 7}. 


Iteration 4: 


1. 


i STN 74 
.on=|{ 1 1-1-2 1 Ss 1 = 3,5) 
a3 1 


JH, DG. 
. 6B =min{—(5/2)/(—3), —2/(—5)} = 2/5, q =7. 
2 Bp =A; 0; 9725/3: 2)? 4 @75)141,0, 3,5)" 


min{—13, 22,0} = -13 <0, s=1. 


= (7/5, 2/5, 9/2, 13/10, 0)!, N = {1,2,3,5,7}, 4 =0+2/5 = 
2/5. 


bid 1 —3 —5 0 
. Ag(R) = (-2- sn) (=) = ( ) piigik') = (S): 
1/2 0 0 


. Gig(R’) = 0. 
. & = (-13)/(-5) = 13/5. 


13. Xp = (—13, 22,0)! — (13/5)(—5, 8, 0)! = (0,6/5,0)!, % = 13/5. 
=173 La 1/3 =1/3. 173 
14. B,' = (= 1 & -1 -s2) = (= —5/3 =i) 
1 1/2 1/2 
15. B ={7, 6,8}, N = {I,2,3,4, 5}. 
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Iteration 5: 


1. min{13/5, 6/5, 0} > 0. 
2. The optimal basic solution and optimal value are 


x* = (0,0, 0,0, 0, 6/5, 13/5,0)!, f* = (1, 5)(6/5, 13/5)! = 71/5. 


20.3 Dual Deficient-Basis D-Reduced Algorithm: Tableau 
Form 


In this section, the deficient basis is incorporated into the dual D-reduced simplex 
algorithm (Sect. 17.2). The new algorithm should be stable, compared with the dual 
deficient-basis algorithm, presented in the previous section, and the related search 
direction corresponds to the entire dual objective gradient. 

Consider the D-reduced problem (17.1). Assume that the order of the deficient 
basis equals | < k < m — | and that the r-indexed row is the datum row (1 <r < 
m). The sets B, R, N, and R’ are defined by 


BeSAhiy osxs iets (J @R=fi,-..,.4}, N=A\B, R'=(1,...,m}\(RU{r}). 
(20.3) 


The initial tableau of (17.1) is 


XB XN f RHS 
Br NR 
Br Nr (20.4) 
ol ol 1 
a ch —1 


which can be converted to the following form by a series of elementary transforma- 
tions: 


XB XN f RHS 
Ne (20.5) 
=T | 


The preceding is called deficient-basis D-reduced tableau, where the mth row is 
the datum row, indexed by r ¢ R, R’. 

Assume that the current tableau is dual feasible, i.e., zy > O. If entries of the 
datum row are no more than zero, then there is no feasible solution to the original 
problem (Lemma 17.1.1). If this is not the case, determine 6 and column index g 
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such that 
B = %4/@q = min{Z;/; |; > 0, j € N} <0. 


Add — times of the datum row to the bottom row to turn Z, to 0. Then the southeast 
entry of the tableau becomes —f, and hence the maximum possible increment of the 
objective value just equals 6 > 0. Under the assumption of nondegeneracy, 6 > 0, 
and hence the objective value strictly increases. 

The determination of the pivot row index depends on whether dg (R’) vanishes. 


(i) Rank-remaining iteration: R’ = Y or Gg (R’) = 0. 
Determine s € argmax{qj,,|t=1,...,k}. If aj,.q < 0, itis clear by @, > 0 
that the tableau corresponds to a basic feasible solution, and hence optimality 
is achieved (taking the g-indexed column as the datum column). If a;,.g > 0, 
take it as the pivot to make the basis change. In such an iteration, the rank of 
the basis remains unchanged. 

(ii) Rank-increasing iteration: R’ 4 @ and G,(R’) F 0. 


Determine row index p € arg maxjeR’ |4jq|. Take apg as the pivot to make a 
basis change. Then the rank of the basis increases to k + 1. 

In either iteration, the elementary transformations do not touch the datum row, 
the bottom row, and the right-hand side, so the resulting tableau is again a dual 
feasible D-reduced tableau. Repeat these steps until the optimality is achieved or 
the infeasibility of the problem is detected. 

The overall steps are summarized into the following algorithm. 


Algorithm 20.3.1 (Dual Deficient-Basis D-Reduced Algorithm 1: Tableau 
Form) Initial: dual feasible deficient-basis D-reduced tableau of form (20.5) with 
B, R, N, R’, 1 <k < m—1. This algorithm solves the D-reduced problem 17.1. 


1. Stop if J = {j ¢ N | @; > 0} = G (infeasible problem). 

2. Determine 6 and column index q such that B = Z/@g = minjes Z;/@;. 

3. If 6 #0, add —£ times of the datum row to the bottom row. 

4. Go to step 8 if R’ = % or ag(R’) = 0. 

5. Determine row index p € arg maxjeR’ |G; q|- 

6. Convert dp, to 1, and eliminate the other nonzeros in the column by elementary 


transformations. 

7. Setk =k + 1, bring q from N to B and p from R’ to R, and go to step 1. 

8. Determine s € argmax{qj,,, |t=1,..., k}. 

9. If dj,.g < 0, compute x, = 1/a@,, Xp = —XgGq(R), and stop (optimality 
achieved). 


10. Convert d;,g to 1, and eliminate the other nonzeros in the column by 
elementary transformations. 

11. Update (B, N) by exchanging j, and q. 

12. Go to step 1. 
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Example 20.3.1 Solve the following problem by Algorithm 20.3.1: 


min f = 3x4 + 2x5 + x6 + 5x7 + 6x8 
S.t. x1 — 6x4 + x5 + 4x6 + 6x7 = 9, 
+x2 + 2x4 — 7x6 — 8x7 + x8 =—6, 
+23 — 4x5 + 2x = 0, 
+3x4 + 2x5 + 3x7 = 3, 
—x4 + 2x5 —4x6 —6x74+3x8 =2, 


xj 20, f=1,...,8. 


Answer Initial tableau is 


Xp x2 x3 x4 X5 X6 xX7 Xg 


1 —6 1 4 6 
1 2 -—7 -8 1 
1 —4 2 
3 2 eS) 0 
=1 2 -4 -6 3 
3 2 1 5 6 


Multiply row 5 by 1/2, and then add —9, 6, and —3 times of row 5 to rows 1, 2, 
and 4, respectively. 


RHS 


Iteration 1: k = 3, B = {1,2,3}, N= A\B, R = {1,2, 3}, RY = {4}. 


1. J = {5, 8}. 

2. B = min{2/1,6/(3/2)} = 2/1, q=5. 

3. Add —2 times of row 5 to the bottom row. 

4. dg(R’) £0. 

5. max{| — l|}, p =4. 

6. Multiply row 4 by —1, and then add 8, —6, 4, and —1 times of row 4 to rows 1, 
2, 3, and 5, respectively. 

7.k=4,B ={1,2,3,5}, N= A\B, R= (1, 2,3,4}, R' =. 
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Xi 3% XB X4 X5 X6 x7 
1 75/2 26 —63 
1 26* 17 46 

1 —18 -24 —48 

-9/2 1 -6 —12 

4 4 9 

4 5 1 


Iteration 2: 


. J = {4,6,7}. 
. B= min{4/4,5/4, 11/9} =1, q =4. 


R’ = 4%. 


rows 1, 3, 4, and 5, respectively. 


. Add —1 times of row 5 to the bottom row. 


. max{—75/2, 26, —18, —9/2} = 26 > 0, s =2. 
. Multiply row 2 by 1/26, and then add 75/2, 18, 9/2, and —4 times of row 2 to 


ll. k=4,B = {1,4,3,5}, N=A\B, R= {1,2,3,4}, RF =@. 


Xx] X2  =X3 XQ x5 X6 
1 75/52 ~71/52 
1/26 1 17/26* 
9/131 —159/13 
9/52 1 159/52 
=9713 18/13 

1/9 


TLE T: 
_p=0,9=6. 


X7 
87/26 
23/13 

—210/13 
—105/26 
25/13 
11/18 


X8 
—105/52 
~17/26 
107/13 
81/52 
—5/13 
113/18 


RHS 


RHS 


—67/18 


. max{—77/52, 17/26, —159/13, —159/52} = 17/26 > 0, s = 2. 


1 
2 
4. R=9. 
8 
0 


times of row 2 to rows 1, 3, 4, and 5, respectively: 
11. k=4,B={1,4,3,5}, N= A\B, R = {1,2,3,4}, FR’ =9. 


Iteration 4: 


1. J = {8}. 
2. B = 113/18, gq =8. 
4. R' =9. 


8. max{—7/2, —1, —4, -3.2}= -—1 <0, s =2. 
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. Multiply row 2 by 26/17, and then add 77/52, 159/13, 159/52, and —18/13 
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xi X2 1X3 X4 %X5 X6 X7 xg RHS 
1 26/17 77/34 125/17 —7/2 
1/17 26/17 1 46/17 -1 
24/17 1 318/17 288/17 —4 
6/17 159/34 1 72/17 —3/2 

—4/17 —36/17 —31/17 1 1 

1/9 11/18 = =113/18 | —67/18 


9. xg = 1, xp = (7/2, 1,4, 3/2)7. The optimal basic solution and optima value 
are 


x* = (7/2,0,4,0,3/2,1,0,1)!, f*=10. 


It might be accepted that if the same approach is used, the search direction 
determined within a subspace of high dimension has better quality than that 
within a subspace of lower dimension. Therefore, it should be desirable to get 
Algorithm 20.3.1 started from k = 0. The associated sets are 


B,R=@, N=A, RF =U,..0,r—1,7+1,...,m), (20.6) 


and the basis matrix Br may be regarded as of zero order. Subsequent steps 
are the same as those in the usual case. The corresponding tableau is called 
zero-order D-reduced tableau; if it is still denoted by tableau (20.5), 7 and Nr 
should be viewed as empty sets and Np the submatrix of A after the r-indexed 
(datum) row deleted. If the bottom row satisfies zy > 0, it is called zero-order 
dual feasible D-reduced tableau. Initially, therefore, a series of rank-increasing 
iterations are usually performed under the condition dg (R’) # 0. 


Example 20.3.2 Solve the following problem by Algorithm 20.3.1, starting from 
the zero-order D-reduced tableau: 


min $f = 2x; +x2 +2x3 + x4 + 4x5 + 5x6 + 2x7, 


s.t. +2x2 —3x5 —4x6 +6x7 =O, 

—2x4 —4x3 +x%6 —x7=-l, 

+3x2 —2x4 +4x5 = 8, 

—3x1 +2x3 —4x4 —2Xx6 =O, 
2x2 — x3 + 2x5 —5x7=-1, 


xj = 0, JH Ayashi. 
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Answer Initial tableau: 


Bal x2 x3 X4 x5 X6 X7 RHS 


2 =3 =4 6 
=) = a oe 
3 2 4 8 
3 —4 —2 
tl D 250 | 2d 


Convert the preceding to the zero-order D-reduced tableau with max{| — 
1], |8],| — 1]} = 8, r = 3. Take row 3 as the datum row (which involves the 
minimum number of nonzeros). 

Multiply the datum row by 1/8, and then add it to row 2 as well as row 5: 


Bal x2 x3 x4 x5 X6 X7 RHS 
2 —3 —4 6 
=2 3/8 -4 -1/4 1/2 1 -l 
3/8 -1/4 1/2 1 
—3 2 —4 —2 
19/8* -1 -—-1/4 5/2 —5 
2. 1 2 1 4 5 2 


wherek = 0, B=%, R=, N =(1,...,7}, R’ = {1, 2,4, 5}, datum row 
r=3. 
Iteration 1: 


1 J = {2,554 9. 

2. B = min{1/(3/8), 4/(1/2)} = 8/3, q =2. 

3. Add —8/3 times of row 3 to the bottom row. 

4. dg(R’) £0. 

5. max{|2], [3/8], |3/8], |19/8|} = 19/8, p=S. 

6. Multiply row 5 by 8/19, and then add —2, —3/8, —3/8, and —1 times of row 5 
to rows |, 2, 3, and 6, respectively. 

7.k=1, B= {2}, R= RU {5} = {5}, N = {1,3,4,5,6, 7}, R’ = {1, 2, 4}. 


Iteration 2: 


1. J = (3,5, 7} 49. 

2. B = min{2/(3/19), (8/3)/(2/19), 2/(15/19)} = 38/15, gq =7. 

3. Add —38/15 times of row 3 to the bottom row. 

4. dg(R’) F 0. 

5. max{|194/19], | — 4/19]} = 194/19, p=1. 

6. Multiply row | by19/194, and then add 4/19, —15/19, 40/19, and —2 times of 
row | to rows 2, 3, 4, and 6, respectively. 
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XI x2 x3 XA x5 X6 x7 RHS 
16/19 4/19 —97/19 —4 194/19* 
—2 —73/19 —4/19 2/19 1 —4/19 
3/19 —4/19 2/19 15/19 1 
—3 2 —4 —2 
1 —8/19 —2/19 20/19 —40/19 
2 2 5/3 8/3 5 2 —8/3 


x1 x2 X3 x4 X5 X6 x7 RHS 
8/97 2/97 = —-1/2 —38/97 1 
—2 —371/97  —20/97 89/97 
9/97 = —22/97 1/2 30/97 1 
=3 2 —4 —2 
1 —24/97 —6/97 —80/97 
2 8/5 11/5 12/5 5 —26/5 


Iteration 3: 
1. J = {3,5,6} 49. 


2. B = min{(8/5)/(9/97), (12/5)/C1/2), 5/(30/97)} = 24/5, q =5. 
3. Add —24/5 times of row 3 to the bottom row: 


x] x2 x3 X4 X5 X6 RHS 

8/97 2/97 —1/2 —38/97 

=) ~371/97 —20/97 89/97 
9/97 -22/97 1/2 ~~ 30/97 1 

—3 2 —4 —2 

1 —24/97 —6/97 —80/97 

2 112/97 319/97 341/97 —10 
4. dg(R’) = 0. 


8. max{—1/2,0} =0 <0. 
9. x5 = 2,xp = (0, pl. The optimal basic solution and optimal value are 


x*=(0,0,0,0,2,0,1)',  f* =10. 


After pivot column index q is determined, there may be choices for taking either 
Dp € R’' or is € R as the pivot row index. In doing so, Algorithm 20.3.1 gives the 
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priority to R’. An alternative is to focus on numerical stability as follows. Define 
C= Maa taly: SP angle Geg|: a ig — MARAT g | Tyee) 


(where ¢, or C2 is set to zero if R’ or R is empty). Then, if ¢ = ¢1, take p € R’, and 
if € = fo, takeis € R. 

Nevertheless, it seems to be preferable to keep the rank of the basis as low as 
possible. To this end, the following variant gives the priority to R rather than R’. 


Algorithm 20.3.2 (Dual Deficient-Basis D-Reduced Algorithm 2: Tableau 
Form) Initial: dual feasible deficient-basis D-reduced tableau of form (20.5) with 
B, R, N, R' 1 <k < m-—1. This algorithm solves the D-reduced problem 17.1. 


1. Stop if J = {j €¢ N | @; > 0} = G (infeasible problem). 
. Determine column index q and 6 such that B = Z,/@g = minjey Z;/@;. 
. If 6 #0, add —6 times of the datum row to the bottom row. 
. Determine s € arg max{dj,.g |t = 1,..., k}. 
. Go to step 8 if dj,,4 < 0. 
. Convert aj, into 1, and eliminate other nonzeros in the column by elementary 
transformations. 
7. Update (B, N) by exchanging j; and qg, and go to step 1. 
8. If R’ = @ or dg(R’) = 0, compute x, = 1/@g, XB = —Xqdq(R), and stop 
(optimality achieved). 
9. Determine row index p € arg maxjeR’ |Gjq|. 
10. Convert dp, to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
11. Setk =k +1, bring q from N to B and p from R’ to R. 
12. Go to step 1. 


NNW N 


Example 20.3.3 Solve the problem in Example 20.3.1 by Algorithm 20.3.2: 


min f = 2x1; + x2 — 2x3 + x4 — 3x5 + x6 + 2x7 — 2x8 


s.t. 2x7 —3x2 + x3 —6x4+%5+ 4x6 + 6x7 = 0, 
5x1 + 6x3 — 2x4 + 10x6 + 8x7 — xg =0, 
—x1 + 8x2 — 4x5 +2xg =0, 
4x1 — 2x3 + 5x4 + 8x5 — 4x6 + 3x7 =1, 


xj 20, fois: 8. 


Answer Begin with the second tableau in the Answer, i.e., 

Iteration 1:k = 3, B = {1, 2,3}, N = A\B, R = {1, 2, 3}, R’ = {4}. 
1. J = {5, 8}. 

2. B = min{2/1, 6/(3/2)} = 2/1, g =5. 

3. Add —2 times of row 5 to the bottom row. 

4, max{—8, 6, -4} = 6,5 =2. 
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x1 x2 


5.8 gp 0 


6. Multiply row 2 by 1/6, and then add 8,4, 1, and —1 times of row 2 to rows 


1,3, 4, and 5, respectively. 
7. B={1,5,3}, N= A\B, R = {1, 2, 3}, R’ = {4}. 


Iteration 2: 


T1671: 


. Gig(R’) #0. 


COMA kRWN eK 


_ 


_ max{|17/6]}, p = 
. Multiply row 4 by 6/17, and then add 10/3, 19/6, 38/3, and —7/6 times of 


x3 


1 


4. 
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x6 x7 
10/3 5/3 
-19/6 —13/3 
—38/3  —52/3 
17/6 23/3 
7/6 4/3 

5 11 


row 4 to rows 1, 2, 3, and 5, respectively. 


Xx] x2 
1 26/17 
6/17 

24/17 

1/17 

—4/17 

5/7 


x3 


X4 
71/34 
159/34 
318/17 
26/17 
—36/17 
38/7 


. B = min{5/(7/6), 11/(4/3))} = 30/7, gq = 6. 
. Add —30/7 times of row 5 to the bottom row. 
. max{—10/3, —19/3, —38/3} < 0. 


x7 
125/17 
72/17 
288/17 
46/17 
SS1/17 
37/7 


_k=4,B={I,5,3,6}, N=A\B, R = {1,2,3,4}, R =O. 


Te 


RHS 


RHS 


RHS 


—44/7 
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Iteration 3: 


. J = {8}. 

. B = 26/7, q = 8. 

. Add —26/7 times of row 5 to the bottom row. 
. max{—7/2, —3/2, —4, —1} <0. 

R' =. x3 =1, Xp = (7/2,3/2, 4, yf. 


aokWN SE 


The optimal basic solution and objective value are 


x* = (7/2,0,4,0,3/2,1,0,1)2, f* = 44/7 +. 26/7 = 10. 


20.4 Dual Deficient-Basis D-Reduced Algorithm 


Assume that the r-indexed row is the datum row. Denote by Bp the basis matrix, 
associated with basic column and row index sets B and R, and so on. It is not 
difficult to show that the deficient-basis D-reduced tableau (20.5) is equivalent to 
the following revised tableau: 


XB XN f | RHS 

I Br Nr 
Ne — Br By Nr (20.7) 
eln —el BB'NR 1 


cl —clBe'NR  -1 


Based on such an equivalence, Algorithms 20.3.1 and 20.3.2 can be revised. But 
only the former’s revision is formulated. 


Algorithm 20.4.1 (Dual Deficient-basis D-Reduced Algorithm 1) Initial: 
B, R, N, R', 1 < k < m—1. BR', Zn = O. This algorithm solves the D- 
reduced problem 17.1. 


. Compute oy = Nl en — Nine Bla 

. Stop if J = {j € N | @; > 0} = Y (infeasible problem). 

. Determine column index q and 6 such that B = Z,/@g = minjes Z;/@j;. 
. If 8 £0, update zy = Zn — Bon. 

. Compute Gy (R) = Ba'ag(R). 

. Go to step 10 if R’ = J or dg(R’) = ag(R’) — Bridg(R) = 0. 

. Determine row index p € arg maxjepR’ |i q|- 

. Update Bz! by (19.10). 

. Setk =k +1, bring g from N to B and p from R’ to R, and go to step 1. 
. Determine s € arg max{dj,,¢ |t = 1,..., k}. 


SGSOANIADUNF WN KH 


_ 
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11. Ifaj,.g < 0, compute xg = 1/@g, Xp = —Xgdq(R), f = CqXq + clits, and 
stop (optimality achieved). 

12. Update By! by (17.20). 

13. Update (B, N) by exchanging j,; and q. 

14. Go to step I. 


The algorithm can get itself started if zy = cy — I Bac B >= O. In the other 
case, methods, presented in Sect. 17.2.1 or Chap. 15, can be utilized to provide an 
initial dual feasible solution, if slightly modified. 

We point out that the so-called reduced dual elimination (Part I) is a convenient 
tool to convert the dual problem to the D-reduced form. The resulting problem can 
be solved by the dual algorithm presented in this section by generating a sequence 
of primal feasible solutions. 


Example 20.4.1 Solve the following problem by Algorithm 20.4.1: 


min f =x, +2x3+3x4+ 5x6 + 2x8, 


S.t. Xx] x5 —2Xx6 +3xg = 1, 
2x1 —4x343x4 —3x6 +x7 —xg = 3, 
—X3 +4xg = 0, 

—2x, +x2 +2x3 -4x4 +46 =-2, 

3x1 +2x6 —2xg= 0, 


xj 20, joHl,...,8. 


Answer The initial tableau is 


nal x2 X3 X4 x5 XG Xe xg |RHS 


1 1 =e 3] 4 
ae a4. g ae fabs 
4 4 
= tf & <4 4 _2* 
3 fs =) 
1 2 3 5 = > 


which is a dual feasible deficient-basis tableau with B = {5,7,2} and R = 
{1, 2, 4}. 

Convert the preceding to a D-reduced tableau: we might as well take row 3 is the 
datum row. Multiply row 3 by —1/3, and then add —4 and 6 times of row 3 to rows 
1 and 2, respectively. 

For this tableau, k = 2, B = {5,7}, R= {1,2}, N = {1,2,3,4,6, 8}, R’ = 
{3, 5}, the datum row: r = 4. Bk = 1.Zy = (1,0, 2, 3,5, 2yt > 0. 
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Ba x2 X3 X4 X5 X6 Xx7 xg RHS 
2-28 3 
£5), “87a: Sd 223 a ee | 
-1 4 
1 -1/2 -1 2 —2 1 
3 2 —2 
1 2 3 5 2 
Iteration 1: 
coy = 172.10 =? 0)": 
FP tA, 
3. B = min{1/1,3/2}=1, q=1. 
A p00 3:5) SS 2100)! S010 3 7 
5. Gq(R) = (0, -5)!. 
6. Gq(R’) = (0,3)! £0. 
7. max{|3|} = 3, p=S. 
8. r= 1/3, v = —(1/3)(0, —5)! = (0, 5/3), dt = (0,0), U =I. 
1 
By'=| 15/3 
1/3 
9.k=3, B={5,7, 1}, R={I,2,5}. N = (2, 3,4,6, 8}, R’ = {3}. 
Iteration 2: 
21-2 3\)/1 
LOy S12: 9.0)! =. (s/o S13 34 15/3 
pee) 
a1 159.973, 578) 
2. J = {4,8}. 
3. B = min{1/2, 2/(2/3)} = 1/2, q =4. 
Ay 99. 39.1.7- OF" = DAT 19 8 393)! 
= (3/4, 7/2, 0, 25/3, 5/3)!. 
1 2 =) 
5. a(R) =| 15/3] | -3] =| -3 
1/3 0 0 
6. aq(R’) =0. 
10. max{—2, —3, 0} < 0. 
ll. X4 = 1/2, %; = —(1/2)(—2,-3,0)! = (1,3/2,0)!. The basic optimal 


solution and optimal value: 


x* = (0,0,0, 1/2, 1,0,3/2, 0), 


ft =3/2. 
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20.5 Deficient-Basis D-Reduced Gradient Algorithm: 
Tableau Form 


It is noted that the search direction —@y, used in Algorithm 20.3.1 or 20.4.1, is 
independent of the current dual iterate. It is imaginable that if the current iterate 
is close to the boundary, the stepsize would become very small, and hence the 
associated change in the dual objective value is negligible. Exploiting the current 
iterate, in this section we modify the search direction so that it is not only ascent 
with respect to the dual objective but also has a “centering” tendency to leave the 
nearby boundary. This is a trick used in the interior-point method. 

Let B, R, N, and R’ be the sets defined by (19.1) and (19.2). The associated 
D-reduced tableau (20.5) represents the following D-reduced problem: 


max y,, 


YR 
I 0 O ZB ( 0 ) (20.8) 
S.t. = = 2 »a+ —a (ee ; 9 > 0. 
(i NE °) _ (") ZN meee 
r 


Let (Zy => 0, Zg = 0) be the current dual feasible solution. Setting 0 <6 « 1, 
introduce 


wt sa, «figes . 
Zyl = Zyle, s=15 Ga FeN, (20.9) 


where Zy is the diagonal matrix, whose diagonal entries are components of Zy. 
In order to create a “centering” direction, we maximize the following auxiliary 
function: 


ees | rT 
max (& ) ZN, 


subject to constraints of (20.8). From the equality constraints, it follows that 


s-1 T -T--1 271 T 72-1 T 
(& ) zw = (n—k) - (@Nzy )y = (Neiy ) i= (Ned; ) YR 


where the first term is constant. But, we do not want that the objective variable y, be 
involved in the maximization of the auxiliary function. By constraints of (20.8), it is 
known that yr + zg = O, and hence if the value of zg does not change, the related 
value of yr does not either. Thus, by neglecting the first three terms of the auxiliary 
function, the problem comes down to the maximization of the following objective: 


max —(Nez7')T yp, 
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subject to constraints of (20.8). The reduced gradient of the preceding program is 
Ayr = —Npiy - (20.10) 
If Ayr, Ay, = 0 is taken, the associated direction in z-space is then 
AZyn = —NE Ayp = NENR Zl, 
Azg = 0. 
Now it is logical to take 
eee ae Vee vee | 
—@n =—On+UNRNRZy , (20.11) 
—@pg = 0 (20.12) 
rather than —@y as the search direction, where the parameter jz > 0 is a weight of 
the modification. 


Denote by a;(R’), j € N, the R’ indexed subvector of aj. The vector —®y 
given by (20.11) with (20.9) has the following property. 


Lemma 20.5.1 Let js > 0 be given. If, for some k € N, it holds that 
Ze=0, a(R')A40, and 273) 40, ViEN, jk, 


then —@, > 0 holds if the 6 is small enough. 
Proof From (20.9) and (20.11), it follows that 


— By = — Op + WR)" Ne iq! = — Oe + a(R) HRI /S+ He SD HR) a;CRY/z;. 


JEN, j#k 
(20.13) 
By the preceding, it is known that the sign of —a@, is determined by 
lax (R')|I°/3 > 0, 
if the 5 is small enough. Therefore, the claim is valid. oO 


In addition, the following result can be shown similarly. 


Lemma 20.5.2 Let 4 > Oand N’ €N be given. If it holds that 
zj =0, 0<aj(R)40,VjEN and 7 40,ViEN\N’, 


then —@, > 0 holds if the 5 is small enough. 
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Based on the preceding lemmas, a sufficiently small 5 ensures that the compo- 
nents, corresponding to zero-valued dual variables (z; = 0), of the new search 
direction —@ are strictly positive. Therefore, the effects of dual degeneracy can be 
avoided. 

It is seen from (20.11) and (20.12) that this search direction yields from multi- 
plying rows of R’ of the D-reduced tableau by respective components of Ayr and 
adding the resulting rows to the datum row. Since the associated components of the 
right-hand side are zero, the whole right-hand side remains unchanged. Besides, the 
bottom row is not touched by these manipulations. Starting from such a D-reduced 
tableau, we proceed by a so-called “outer iteration,” consisting of a series of “inner” 
D-reduced rank-increasing iterations and terminating when R’ = or dg(R’) = 0; 
if the optimality conditions are not satisfied yet, i.e.,Z, = 0, @, > Obuta,(R) £ 0, 
the next outer iteration is carried out. 

An outer iteration may start from any dual feasible deficient-basis D-reduced 
tableau, including the zero-order tableau, although it seems favorable to take the 
D-reduced tableau associated with the sets 


B=B\{j,|teT}, R=R\{i;|t€T}, T= {t=1,...,k|Gi,,q > 0}. 
(20.14) 


This will separate the rows and columns corresponding to the negative components 
of the current primal feasible solution from the basis. Another scheme is to let a 
part of them leave the basis, e.g., to separate the rows and columns corresponding 
to the negative components of larger magnitude. In fact, under the assumption of 
Lemma 20.5.2, —@;, > O holds for all t € T if 5 is small enough. Therefore, 
the zero Z;,, t ¢ T, have nothing to do with the determination of the stepsize and 
the related index. However, in case zy is degenerate, a positive stepsize is still not 
guaranteed if a,(R) Z 0 and it holds for some j ¢ N that z; = 0, @; > 0. 
The respective algorithm can be obtained by modifying Algorithm 20.3.1. 


Algorithm 20.5.1 (Deficient-Basis D-Reduced Gradient Algorithm: Tableau 
Form) Given uz > 0, 6 > 0. Initial: dual feasible deficient-basis D-reduced tableau 
of form (20.5), associated with B, R, N, R’, 1 < k < m—1. Thisalgorithm solves 
the D-reduced problem 17.1. 


1. Cover nonbasic components of the datum row by 
ar = ot — wey!) WE Ne. 
. Stop if J = {j € N | @; > 0} = Y (infeasible problem). 
. Determine column index q and 6 such that B = Z,/@g = minjes Z;/@j;. 
. If 6 40, add —6 times of the datum row to the bottom row. 
. Go to step 9 if R’ = G or dg(R’) = 0. 
. Determine row index p € arg maxjepR’ |Gjq|- 
. Convert Gp, to 1, and eliminate the other nonzeros in the column by elementary 
transformations. 
. Setk =k +1, bring g from N to B and p from R’ to R, and go to step 2. 


NYDN WNY 


[oe] 
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9. Determine s € argmax{dj,.¢ |t = 1,..., k}. 

10. If aj,., > 0, update B, R by (20.14), and go to step 1. 
11. Compute x, = 1/@g, Xj, = —Xqi,,g, t= 1,...,k, 
12. Stop (optimality achieved). 


Note There are two layers of iterations in the preceding algorithm. The inner one 
consists of steps 2 to 7. 


Example 20.5.1 Solve the following problem by Algorithm 20.5.1: 


min f =x; +2x2+ x3 + 7x4 + 3x5 + x6, 


s.t. —3x1 —X3 —3x5 +2x6 = —l, 
—2x, —x2 +2x3 —6x4 +6x5 —x6 = —2, 
+3x2 —2x3 +8x5 +4x%5 = 0, 

—6x1 +9x4 —3x5 = 3, 


xj>0, jHl,...,6. 


Answer 4 = | and 6 = 1/10 are taken. Initial tableau: 


xX] x2 X3 X4 x5 X6 -RHS 

—3 —l —3 2 |-1 

—2 -l 1 -6 6 —-1 |-2 
2 8 4 

—6 9 —3 3% 


Convert the preceding to a D-reduced tableau: max{| — 1], | — 2], |O|, |3|} = 
3, r = 4, and take row 4 as the datum row. 

Multiply row 4 by 1/3, and then add 1 and 2 times of row 4 to rows | and 2, 
respectively. 


; xX] x2 X30 °=«X4 XS X6 | RHS 
—5 -1 3. -4 2 
-6 -l 1 4 -1 | 
3 -2 8 4 | 
—2 3 -l 1 


Iteration 1: 
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Set Oy = (—2, 0,0, 3, -1, 0) — (1, 1/2, 1, 1/7, 1/3, DNENe 
= (—1208/21, —71/3, 529/42, 124/7, —345/7, —841/42) 
to as the left part of row 4: 


Xx] x2 X3 X4 X5 X6 RHS 
=5 -1 3 —4 2 
—6 -1 1* 4 -1 
3 —2 8 4 
—1208/21 —71/3 529/42 124/7 —345/7 —841/42) 1 
1 2D: 1 7 3 1 
2,3 = 3,4) 40. 


3. B = min{1/(529/42), 7/(124/7)} = 1/(529/42), ¢q =3. 

4. Add —8 times of the datum row to the bottom row. 

5. dg(R’) # 0. 

6. max{| — 1], |1],| —2]}= 2, p =2. 

7. Add 1, 2, —529/42, —1 times of row 2 to rows 1,3,4,5, respectively. 
8. B= {3}, R= {2}, N={1,2,4,5, 6}, R’ = {1, 3}. 


x1 x2 x3 x4 X5 X6 RHS 
-l1 -1 3 1 
—6 -1 1 4 -1 
—12* 1 16 2 
379/21 —155/14 124/7 —299/3,  —52/7 1 


2945 /529 2052/529 2959/529 159/23 1370/529| —42/529 


Iteration 2: 


2,.J={1,4 49. 

3. B = min{(2945/529)/(379/21), (2959/529)/(124/7)} 
= (2945/529)/(379/21), gq =1. 

. Add —f times of the datum row to the bottom row. 

. Gg (R’) # 0. 

. max{| — 11|,|— 12|} = 12, p =3. 

. Multiply row 3 by —1/12, and add 11, 6, —379/21, —2945/529 times of row 3 
to rows 1, 2, 4, 5, respectively. 

8. B = {3,1}, R = {2,3}, N = {2,4,5, 6}, R’ = {I}. 


NAYDAUNA 


Iteration 3: 


2.J={4 49%. 
3. B = (49/379)/(124/7), q = 4. 
4. Add —8 times of the datum row to the bottom row. 
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Xx} x2 x3 xX X5 X6 RHS 

—23/12 3% —44/3 —5/6 

—3/2 1 —4 —2 
1 —1/12 —4/3 —1/6 
—2411/252 124/7 —4763/63 —557/126 1 

5529/758 49/379 6477/172 1850/379 | —147/379 
5. dg(R’) # 0. 
6. max{|3|}, p = 1. 


7. Multiply row 3 by 1/3, and then add —124/7 and —49/379 times of rows 4 and 
5, respectively. 


8. B = (3, 1,4}, R= {2,3, 1}, N= (2,5, 6}, R’ =9. 


XxX] x2 X3 x4 x5 X6 RHS 
—23/36 Ie. S419 —5/18 
=3)0 =4 -2 
1 —1/12 -4/3 -1/6 
7/4 ll 1/2 1 
1760/239 3477/91 1307/266| —49/124 


Iteration 4: 


2. J =(2,5,6} £9. 


3. B = min{(1760/239)/(7/4), (8477/91) /11, (1307/266)/(1/2)} 
= (3477/91)/11, q =5. 
4. Add —8 times of the datum row to the bottom row. 


xX] x2 X30 (X4 x5 X6 RHS 
—23/36 1 —44/9 —5/18 
—3/2 1 —4 —2 
1 —1/12 —4/3 -1/6 
7/4 11 1/2 1 
2027/1577 629/198 | —383/99 
5. Gq(R’) =0. 


9. max{—44/9, —4, —4/3} < 0. 


11. x5 = 1/11, x8 = A/1DG, 4/3, 44/9)1 a (4/11, 4/33, 4/9)?. 
12. The optimal basic solution and optimal value are 


x* = (4/33, 0, 4/11, 4/9, 1/11, 0)!, f* = 383/99. 
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Example 20.5.2 Solve the following D-reduced problem by Algorithm 20.5.1: 


min f =3x4+%6 + 2x7, 


S.t. x] —2x4 +5x5 —5x6 +2x7 = 0, 
+x2 —3x4 +3x5 —2x6 +x7=0, 

+x3 +x4 —4x5 +6x6 —3x7 = 0, 

—6x4 +3x6 +x7=0, 


+x4+4x5 +2x6 -—x7 = 1, 
apEO. fF Hl ecey 7. 


Answer 4 = | and 6 = 1/10 are taken. Initial tableau: 


X} x2 X3 X4 x5 X6 x7 | RHS 
1 —2 5 -—5 2 
1 —3 3 -2 1 
1 1 —-4 6 -3 
—6 0 3 1 
1 4 2 -l 1 
3 0 1 2 


Row 5 is the datum row: r = 5. B = {1,2,3}, R = {1,2,3}, N = 
{4,5,6, 7}, R’ = {4}. 
Column 5 is the datum: aj,5 = 5 <0, a25=3>0, T = {1,2}. 
Reset B = {3}, R= {3}, N= {1,2,4,5, 6,7}, R’ = {1, 2, 4}. 
Iteration 1: 
1 -—25-52 
1. Nr = 1-33-21 
-6 31 
Cover nonbasic components of row 5 by 
ot = (0,0, 1,4, 2, —1) — C10, 10, 1/3, 10, 1, 1/2)NE NR 
= (—166/3, —75/2, 1399/6, —2311/6, 2095/6, —452/3): 


xX] x2 x3 X4 X5 X6 x7 RHS 
1 —2 5 —5* 2 
1 3 3 —2 1 
1 1 —4 6 3 
—6 0 3 1 
—166/3 —75/2 1399/6 —2311/6 2095/6 —452/3) 1 
| 0 1 2 


2. J ={4,6} 490. 
3. B = min{3/(1399/6), 1/(2095/6)} = 6/2095, q = 6. 


20.5 Deficient-Basis D-Reduced Gradient Algorithm: Tableau Form 559 


. Add —f times of the datum row to the bottom row. 

. Gg (R’) # 0. 

. max{| — 5], | — 2|, |3|}=5, p= 1. 

. Multiply row 1 by —1/5, and then add 2, —6, —3, —2095/6, and —1 times of 
row | to rows 2, 3, 4, 5, and 6, respectively. 

8. B = {3,6}, R = {3,1}, N = {1,2,4,5, 7}, R’ = {2, 4}. 


NDAMNA 


x] x2 4X3 X4 xe XS XG RHS 
-1/5 2/5 -l1 1 —2/5 
—2/5 1 -11/5 1 1/5 
6/5 1 -—7/5 2 —3/5 
3/5* —36/5 3 11/5 
29/2 —75/2 187/2 —36 —11 1 


332/2095 45/419 1397/599 1102/999 = 5094/2095) —6/2095 


Iteration 2: 


2,.J={1,4 49. 

3. B = min{(332/2095) /(29/2), (1397/599) /(187/2)} 
= (332/2095)/(29/2),q = 1. 

. Add —f times of the datum row to the bottom row. 

. Gg (R’) # 0. 

. max{| — 2/5], |3/5|} = 3/5, p = 4. 

. Multiply row 4 by 5/3, and then add 1/5, 2/5, —6/5, —29/2, and —332/2095 
times of row 4 to rows 1, 2, 3, 5, and 6, respectively. 


NYDHAMNA 


8. B = {3,6, 1}, R= (3, 1,4}, N = {2,4,5, 7}, RB’ = {2}. 
x] x2 X3 X4 X5 X6 x7 RHS 
2 1 1/3 
1 -7# 3 5/3 
1 13 —4 —5 
1 -12 5 11/3 
—75/2 535/2  —217/2 —385/6 1 
15/29 38/29 217/145 74/29) —2/145 


Iteration 3: 


J ={4 4G. 

. B = 38/29)/(535/2), q =4. 

Add —f times of the datum row to the bottom row. 

. Gg (R’) # 0. 

. max{| — 7|}, p =2. 

. Multiply row 2 by —1/7, and then add 2, —13, 12, —535/2, and —38/29 times 
of row 2 to rows 1, 3, 4,5, and 6, respectively. 
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8. B = {3,6, 1,4}, R= (3, 1,4,2}, N = {2,5,7}, RB’ =9. 


XxX] x2 X3 X4 x5 X6 XT RHS 
2/7 -6/7 1 1/7 
-1/7 1-3/7 —5/21 
13/71 11/7 —40/21 
1 -12/7 1/7 17/21 
5/7 43/7 —10/21 1 
75/107 217/107 920/321] —2/107 
Iteration 4: 
2. J = {2,5} 4G. 


3. B = min{(75/107)/(5/7), (217/107) /(43/7)} = (217/107)/(43/7), q =5. 
4. Add —8 times of the datum row to the bottom row: 


x4 x2 x3 (M4 x5 X6 x7 RHS 
—2/7 -6/7 1 1/7 
—1/7 1-3/7 —5/21 
13/7 1 11/7 —40/21 

1 —12/7 —1/7 17/21 
5/7 43/7 —10/21 1 
20/43 130/43 | —15/43 

5. Gqg(R’) = 0. 


9. max{11/7, —6/7, —1/7, —3/7} = 11/7 > 0, s =1. 
10. T = {1}, B= {6, 1,4}, R = (1,4, 2}, N = {2,3,5,7}, R’ = {3}. 


Iteration 5: 


1. Ne = (13/7, 1, 11/7, —40/2). Cover nonbasic components of row 5 by 
at = (5/7, 0, 43/7, — 10/21) — (43/20, 10, 10, 43/130)NENp 
= (—22540/423, —12067/415, —7554/191, 15649/285): 


Xx] x2 x3 x4 x5 X6 x7 RHS 
—2/7 —6/7 1 -1/7 
-1/7 1 —3/7 —5/21 
13/7 1 11/7 —40/21* 
1 —12/7 —1/7 17/21 
—22540/423 —12067/415 —7554/191 15649/285 1 
20/43 130/43 | —15/43 


y 
. B = (130/43) /(15649/285), q = 7. 
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. Add —8 times of the datum row to the bottom row. 

. Gg (R’) # 0. 

. max{| — 40/21|}, p =3. 

. Multiply row 3 by —21/40, and then add 1/7, 5/21, —17/21, —15649/285, and 
—130/43 times of row 3 to rows 1, 2, 4, 5, and 6, respectively. 

8. B= {6,1,4,7}, R= {1,4, 2,3}, NM = {2,3,5}, R’ =W. 


NAN 


x1 x2 Xx3 X4 X5 X6 X7 RHS 
17/40 —3/40 —39/40 1 
—3/8 217°? i —5/8 
~39/40 ~21/40 ~33/40 1 
i —37/40 17/40 21/40 
100463/401854+19713/78851 23/4 1 
1397/411 658/411 2624/1205 —767/1899 


Iteration 6: 


2. J ={2,5}49. 

3. B = min{(1397/411)/(100463/401851), (2624/1205) /(23/4)} 
= (2624/1205) /(23/4), g =5. 

4. Add —8 times of the datum row to the bottom row. 


x1 x2 X30 «X4 X5  X6 RHS 
—17/40 —3/40 —39/40 1 
—3/8 —1/8 1 —5/8 
—39/40 —21/40 —33/40 
1 —37/40 17/40 21/40 
100463/401854 19713/78851 23/4 1 
76/23 39/23 —18/23 
5. Gq(R’) =0. 


9. max{—39/40, 21/40, —5/8, —33/40} = 21/40 > 0, s = 2. 
10. T = {2}, B = {6,4,7}, R= {1, 2,3}, N = {1,2,3,5}, R’ = {4}. 


Iteration 7: 


x1 x2 X3 XA x5 x6 RHS 
—17/40 —3/40 —39/40 1 
~3/8 ie id 5/8 
~39/40  —21/40 ~33/40 
1 —37/40* 17/40 21/40 
—3379/222  4571/319 —3823/569 —3498/1561 1 
76/23 39/23 —18/23 
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2. 


3 
4 
5. 
6 
7 


J={23490. 
B = (16/23)/(4571/319), gq =2. 


. Add —f times of the datum row to the bottom row. 


Gig(R') £0. 


. max{| — 37/40|}, p = 4. 
. Multiply row 4 by —40/37, and then add 17/40, 3/8, 39/40, —4571/319, and 


—76/23 times of row 4 to rows 1, 2, 3,5, and 6, respectively. 


8. B= {6,4,7,2}, R= {1,2,3,44, N={1,3,5}, RF =@. 
xX] x2 X3 x4 x5 X6 x7 RHS 
17/37 —10/37 —45/37 1 
—15/37 -11/37 1 —31/37 
39/37 36/37 51/37 1 
—40/37 1 9 -17/37 21/37 
3957/14641 —793/5868 218/37 1 
2471/704 6039/1861 910/1761 —997/984 
Iteration 8: 
2. J ={1, 5} 49. 
3. B = max{(2471/704)/(3957/14641), (910/1761)/(218/37)} 


4. 


= (910/1761)/(218/37), g =S. 
Add — times of the datum row to the bottom row. 


xX] x2 x3 x4 x5 X6 X7 RHS 
—17/37 —10/37 —45/37 1 
—15/37 —11/37 1 —31/37 
—39/37 —36/37 —51/37 1 
—40/37 1 —17/37 —21/37 
3957/14641 —793/5868 218/37 1 
380/109 355/109 —120/109 
. Gig(R') = 0. 


9. max{—45/37, —31/37, —51/37, —21/37} < 0. 


12. 


. ¥5 = 37/218, Xp = (37/218)(45/37, 31/37, 51/37, 21/37) - 


= (45/218, 31/218, 51/218, 21/218)!, 
The basic optimal solution and optimal value: 


x = (0, 21/218, 0, 31/218, 37/218, 45/218, 51/218)!, f = 120/109. 
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20.6 Deficient-Basis D-Reduced Gradient Algorithm 


It is not easy to convert the tableau Algorithm 20.5.1 to the revised version because 
of the difficulty to update Br: in step 10. The tableau Algorithm itself seems to be of 
practical value. However, the situation will be different if every outer iteration starts 
from the zero-order tableau instead. This idea is realized in the following algorithm. 


Algorithm 20.6.1 (Deficient-Basis D-Reduced Gradient Algorithm) Given p > 
0,5 > 0. Initial: z > 0. This algorithm solves the D-reduced problem 17.1. 


1. Set k = 0, and give B, R, N, R’ by (20.6). 

2. Compute wy = Nl em _ UNE NR), and set Oy = wy. 

3. Stop if J = {j € N | @; > 0} = G (infeasible problem). 

4, Determine column index qg and 6 such that 6 = Zy/@g = minjey Zj/@;. 
5. If 6 £0, update Zy = Zy — Oay. 

6. Compute ag(R) = Ba; (R) and Gg (R") = ag(R’) — Brag (R). 
7. Go to step 12 if R’ = M or ag(R’) = 0. 

8. Determine row index p € arg maxjeR’ |G ql. 

9. Update Bz’ by (19.10). 
10. Setk =k + 1, bring g from N to B and p from R’ to R 
11. Compute oy = wy — N} (Bz! op), and go to step 3. 


12. Determine s € argmax{qj,,, |t=1,..., k}. 
13. Go to step 1 if aj, g > 0. 
14. Compute x, = 1/@g, XB = —Xqdq(R), f= CqXq + clip. 


15. Stop (the optimality achieved). 
An alternative way is to utilize, to a larger extent, current information to form the 
datum row r. Only the first iteration starts from 


w= Ate, — pALARz!, (20.15) 


where Ap denotes the matrix, resulting from A by deleting row r. Assume that at 
the end of some subsequent inner iterations, the basic column and row index sets 
are B and R, not satisfying the optimality condition, the datum row @ is 


op=0, ON =n - ni (B;" en) ; 


and the basic row matrix A R 1s 


Br=I, Nr=By'Nr. 
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Then the datum row @ for the next outer iteration is formed by combining the 
preceding two items as follows: 
As al { s-1 _ s-l =1y;.5=1 
OBR = OB — LBpARZ =U ZB + Bp NRZn > 
On = On — uNPARz! 
= on — NA Ba on — UNA (25! + By NeZy') 
= oy — NEBR" (op — On). 
Therefore, we have the following variant of Algorithm 20.6.1. 


Algorithm 20.6.2 (Variant of the Dual Reduced Gradient Algorithm) Same as 
Algorithm 20.6.1 except steps I and 2 replaced by 


1. Compute w by (20.15), and set ® = o. 
2. Set® = aw, k = 0; with sets B, R, N, and R' are determined by (20.6), and 
steps 13-15 replaced by 


13. If ap,q < 9, compute xg = 1/@g, XB = —XqQq(R), f= CqXq + ata and 
stop (optimality achieved). 
14. Compute v = —Z,' — Ba’ Nriy'. 


15. Compute oy = on — NI BA" op —v), WB =v. 
16. Go to step 1. 


It is noted that the preceding variant solves two extra systems in each iteration. 


Reference 


Pan P-Q (2005) A revised dual projective pivot algorithm for linear programming. SIAM J Optim 
16:49-68 


Chapter 21 ® 
Face Method with Cholesky Factorization = xz 


The deficient-basis method performs well, but the only drawback is that the number 
of columns of the basis increases in the solution process, weakening its benefits, 
which lets us believe that it is impossible to overcome degeneracy by moving vertex 
by vertex. 

Based on the Cholesky factorization, the face method presented in this chapter 
will no longer move vertex by vertex, but face by face. Therefore, it would bring 
about the hope of a complete escape from the degeneracy “nightmare.” In the 
subsequent sections, we address such themes as the steepest descent direction, 
update of the solution, face contraction, optimality test, face expansion, face 
algorithm supported by the Phase-I based on the single-artificial-variable approach, 
and related computational results. Before making our comments on the face method 
in the final section, we spend two sections on its variants: the affine face algorithm 
and generalized face algorithm. 

Consider the reduced problem (16.1), i.e., 


mn (21.1) 
st. Ax=b, xj >0, j=Hl,...5n, 
where AE ROTDX@+D DER"! rank A=m+1,m <n, any = Anti = 
—€m+1. The assumption on the rank of A will be dropped later. 
Assume that Be RO"t)*k) NE ROMt+DX@—-K+D ig a partition of A, where 


rank B=m+1, m+1<k<n+l. 


The matrices B and N are called face matrix and nonface matrix, and the associated 
variables are face variables and nonface variables. Without confusion, denote the 
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associated index sets by B and N, respectively, i.e., 
B=(l,...,k-l1,n+]}, N={l,...,n+]}\B. 


Note that n + 1 is the last index of B. 
Thereafter, it is assumed that the kth (last) column of the face matrix equals the 
(n + 1)th (last) column of A, Le., 


Bex = —em41- (21.2) 
The face, associated with the face matrix B, is defined by 
Pp={xeEA|Bxg=b; x5 >0, s=1,...,k-—1, xy =O}. (21.3) 


A point/solution on the face is called face point/solution. Thus, the zero-dimensional 
face has the unique face point—vertex. 


Definition 21.0.1 A face is a level face if the objective value is constant over it. A 
level face is an optimal face if the related objective value equals the optimal value. 


Assume that x is the current face point on Pg, ie., x € Pg. As x is uniquely 
determined by xz, the two will not be strictly distinguished hereafter; for simplicity, 
Xp € Pe is often used in place of x € Pg. 


21.1 Steepest Descent Direction 


Firstly, we determine the search direction. Consider the following subprogram: 
min f, 
s.t. Bxg =b, (21.4) 
xp>0, n+1¢B, 


which actually minimizes the objective over the face Pp. 
Introduce k x k orthogonal projection matrix, i.e., 


P=1—B'(BB')-'z. (21.5) 


Then, the orthogonal projection of the objective gradient, e;, of the subprogram onto 
the null space of B is 


Ap = Per =ex—By, BB! ¥ = emi, (21.6) 


and — Az is a desirable search direction. In fact, we have the following result. 
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Proposition 21.1.1 The vector Ag satisfies BAz = 0. 
Proof By (21.6), it is easy to verify validity of the equality. oO 


Proposition 21.1.2 The following statements are equivalent: 
(i) Ag 4 0; (ii) Ax > 0; (ili) ex ¢ range(B"). 


Proof Premultiply the first expression of (21.6) by AN Then combining the result 
and Proposition 21.1.1 gives 


AT An = Ak. 
Therefore, statements (i) and (ii) are equivalent. If Az = 0, it is known by the fist 


expression of (21.6) that eg = B'S, 1.€., ee € range(B!), and hence (ili) results in 
(i). Conversely, if eg € range(B!), then there is some vector u such that 


Gea Bow. (21.7) 
Premultiplying the preceding by B together with (21.2) gives 
m1 = BB  u. 


Consequently, u = y follows from the preceding, the second expression of (21.6), 
and the full row rank of B. Furthermore, it holds by (21.7) that 


ek = B'S. 


Thus substituting the preceding into the first expression of (21.6) gives Ag = 0. 
Therefore, (i) and (iii) are equivalent. oO 


Proposition 21.1.3 [f Ag + 0, then it holds that 
T = 
e, Ap = Ax > 9, (21.8) 
and 
T T 
—e& Ag/||Aall < ea v/|lvll, VOAv e Null (B). (21.9) 


Proof The expression (21.8) can be derived directly from Proposition 21.1.2. 
For P defined by (21.5), it is verified that 


Pu=v40, VOAvENUI(S). (21.10) 
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Furthermore, it follows from the Cauchy inequality that 
T 
(—Pex) (Pv) < ||Pex|l|| Pull. 
Multiplying the preceding by 1/|| Pu|| gives 


(— Pex)! (Pv)/|| Pull < | Pexll. 


(— Pex)! (Pv)/|| Pull < (Pex)! (Pex)/||Pexll- 


Consequently, the preceding together with P? = P, pl = P, (21.10), and Ag = 
Pex gives (21.9). oO 


According to Proposition 21.1.3, vector —Ag + 0 forms the most-obtuse-angle 
in the null space of B with the objective gradient e; of the subprogram and is hence 
the steepest descent one. In this sense, taking the vector as a search direction is the 
“best” choice. 

It is noted that the computation of Ag mainly lies on solving the (m+ 1) x (m+1) 
system 


BB  y = —ema. (21.11) 


Since B is of full row rank, there exists the Cholesky factorization (Golub and Van 
Loan 1989) 


BB! =LL!, (21.12) 
where L is nonsingular lower triangular. Therefore, system (21.11) becomes 
LE p= e554, (21.13) 
which can be put into the following two triangular systems: 


Lv = —€m+1, y=v.z 


The solution to the first system is readily available, ie., v = —(1/v)em+1, where v 


is the (m + 1)th diagonal of L. Consequently, there is only one triangular system 
left to be solved, 1.e., 


LTy = -(1/v) ema. (21.14) 
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21.2 Updating of Face Solution 


The new iterate is defined by the following line search scheme: 


xXp=Xp—adsz, (21.15) 


where @ is a stepsize to be determined. If the index set 


J={jEBlA;>0,j Ant) 


is nonempty, determine @ and p such that 


a = Fp/Ap = min t)/Aj = 0, (21.16) 


which is the maximum possible stepsize for the new iterate remaining within the 
feasible region. If x; = 0 for some j € B, j #n-+ 1, then face point x is said to 
be degenerate. In this case, the stepsize a could vanish, and the resulting iterate is 
actually not “new” but the same as the old one. Note that whether x is degenerate 
only depends upon the first k — 1 components of xg. 


Lemma 21.2.1 Assume that xg € Pg and Ag # 0. 


(i) 
(it) 


If J # @, then Xp € Pp is a boundary point. The objective value does not 
increase and strictly decreases if xg is nondegenerate. 
If J = G, the original problem is lower unbound. 


Proof 


(i) 


(ii) 


Az # Oand J ¥ @ together ensure that (21.16) is well-defined. As xg € Pp, 
it is known from Proposition 21.1.1 that <g € Pg; furthermore, it is clear that 
Xp = 0, and hence X is a boundary point, associated with the new objective 
value 


A 


f = f —adg. (21.17) 


It is known from (21.16) that a > 0. Besides, from Ag # 0 and Proposi- 
tion 21.1.2, it follows that Ay > 0, and hence f< < f. If g is nondegenerate, 
then a > 0, leading to f & f. 

When J = Q, it is clear that Xg € Pg holds for any a > O. It is then known 
by (21.17) that 


f— -0oo as a> ow. 
Therefore the objective value is lower unbounded over Pg, hence over the 


feasible region. 
oO 


570 21 Face Method with Cholesky Factorization 
21.3 Face Contraction 


Assume that Ag 7 0 and that the new iterate X, was determined by (21.15) with 
Xp = 0. Updating B and N accordingly gives the face and nonface index sets, 
associated with the new iterate, i.e., 


B=B\{p}, N=NU{p}. 
The associated face matrix B ¢ R"+)**—D results from the old B by dropping 
the p-indexed column. Then we have the following result. 


Proposition 21.3.1 [frank B = m + 1, then rank B=m+t1. 


Proof It is clear that rank B <m+1.Itis known by (21.16) that A, 4 0 and by 
Proposition 21.1.1 that 


y > Ajaj = BAg =0, 
jcB 


and thus 


ap =—(l/Ap) D> Ajaj. 


SEB, j#p 
The above equality implies that a, € range B. Therefore, rank B < m + 1 leads to 
rank B < m + 1, as contradicts the assumption. Therefore rank B=m +1. oO 


The preceding proposition implies that the new face matrix Bis again of full row 
rank, and hence there exists the Cholesky factorization 


BB! LL", 
Saunders (1972) offers a procedure to obtain the new factor from its predecessor, as 


is put into the following algorithm. 


Subalgorithm 21.3.1 (Subalgorithm: contraction) Initial: the Cholesky factor L of 
BB". This algorithm finds the Cholesky factor i of BB!. 


(1) Solve lower triangular system Lu = ap. 


(2) Compute y = /1 — |lu||?. 


(3) Determine Givens rotations J,1,...,N 1, where J; is in the (m + 2, , )-plane, 


such that 
0 
Mee dnst (“) =( : 
y 1 
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tt it ; 
(4) Compute Nj... Jin+1 =| ‘7 }. toobtainL € Rint) xt), 
a 


(5) Return. 


21.4 Optimality Test 


In case of Ag = 0, the search direction is not well-defined by (21.6). In particular, 
it is the case when k = m + 1, as the projection matrix is a zero one. 


Lemma 21.4.1 Assume that Ap is defined by (21.6). If Ag vanishes, then Pg is a 
level face, and vice versa if0 < xp € Pg. 


Proof Assume that y satisfies the second expression of (21.6). From Ag = 0 and 
the fist expression of (21.6), it follows that 

ee = By. 
Transpose the two sides of the preceding equality, and postmultiply the result by 
any xg € Pg to obtain 


ef xp — 5! Bxp, 


which, combined with and Bxgz = b, gives 


f=y". 


This implies that the objective value is constant over Pg, and hence Pz is a level 
face. 

Now assume that Pg is a level face and that 0 < xg € Pg with Ag ¥ 0. In the 
case when Ag < 0, it is known by (ii) of Lemma 21.2.1 that the objective value is 
lower unbounded over Pz, which contradicts that Pg is a level face: when Ag < 0, 
since xg is nondegenerate, it is known from (i) of Lemma 21.2.1 that the objective 
value, associated with *g € Pg given by (21.15) and (21.16), strictly decreases, as 
also contradicts that Pg is a level face. Therefore, Ag vanishes. oO 


Now consider the dual problem of (21.1) 


T 
B ZB ek (21.18) 
e + = ; : > 0, = 0. 
Ss (ir)> 3) (“) ZB, ZN Zn+1 
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Lemma 21.4.2 Assume that Xg € Pg and Ag = 0. If iy = —N’¥ > 0, then 
(xB,Xn) and (Zp, Zn, y) are a pair of primal and dual optimal solutions, where 
Xn = 0, Zp = 0, and Pz is an optimal face to (21.1). 

Proof (xg, Xn) (xn = 0) is clearly a feasible solution to (21.1). On the other hand, 
combining (21.6) and Ag = 0 gives Bl y = ex. Therefore, it is easy to verify that 
(ZB, ZN, y) is a feasible solution to (21.18), satisfying the complementarity condi- 
tion, and hence is a pair of primal and dual optimal solutions. By Lemma 21.4.1, 
Ag = O implies that Pp is a level face. It includes the optimal solution and hence is 
an optimal face. Oo 


According to Proposition 21.1.2, Ag = O if and only if A; = 0. Therefore 
condition Ag = 0 may be replaced by A; = 0 simply. 


21.5 Face Expansion 


Assume Ag = 0 and zy = —N T y % 0. In this case, optimality of the level face Pg 
cannot be asserted, but a column index g can be determined such that 


€ argmin Z;. 
q & yeN Zi 
Accordingly, update the face and nonface index sets by 
B=BU{q}, N=N\{q}. (21.19) 


As a result, the number of columns of the face matrix will increase by 1. 

Thus, the g-indexed column is added to the face matrix. Denote the resulting new 
face matrix by Bee RmtDx (K+) which is still of full row rank. The corresponding 
updating of the Cholesky factor is simple (Golub 1965), stated as follows. 


Subalgorithm 21.5.1 (Subalgorithm: expansion) Initial: the Cholesky factor L of 
BB". This algorithm finds the Cholesky factor L of BB. 
Determine Givens rotations Nj,..., Jm+1, where J; is in the (i,m + 2)-plane, 


such that 
T FT 
L L 
Bera ...N — ’ 
— (3) @ 


which gives L € ROW+Dx m+), 


i 
VY 


The validity of the preceding subalgorithm is verified by 


EL! = LL! + aja} = BB! + a,a) = BB. 
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Another scheme, seeming to be attractive, is to use 
B=BU{jEN|z <0}, N=A\B 


rather than (21.19), though adding two columns to the face matrix will complicate 
updating of the Cholesky factor. 


21.6 Face Algorithm 


Any submatrix B € RtDxk with m +1 <k <n+1andrank (B) =m+1 
can be taken as an initial face matrix although we prefer the maximum one, i.e., 
k=n+1, B=A. 

Now drop the assumption that A is of full row rank. Assume that an orthogonal 
matrix [Q1, Q2] «e R@+)*@+) and a permutation M «¢ R°"+)*(™+) have been 
found such that 


AT =(01.001( 5); 


where R € R’*" is a nonsingular upper triangular. Then L = R! is the initial 
Cholesky factor. In fact, rank A = r, and the last m — r + 1 equations of 


Tl Ax = M'b are redundant: if the original notation is still used to denote the 


problem, resulting from dropping these redundant rows, then Al = Q1R, and the 
wanted Cholesky factorization is 


AA! = Rg! g,R = LL, (21.20) 


The overall steps can be put into the following algorithm. 


Algorithm 21.6.1 [Face algorithm with Cholesky factorization] Initial: (B, NV), 
m+1< k < n+1, feasible solution x, and the Cholesky factor L of BB!. 
This algorithm solves the reduced problem (21.1). 


1. Solve upper triangular system LTy = —(1/v)em+1, where v is the (m + 1)th 
diagonal of L. 

2. Ifk >m-+41, compute Ag = ex — B'S. 

3. Goto step 10ifk =m+1lorAgz=0. 

4. StopifJ={je B\|A;>0, jAn+1=¢9. 

5. Determine stepsize a and index p such that a = X¥)/Ap» = minjey X;/Aj. 

6. Ifa £0, update xp = X—p — Ag. 

7. Update (B, N) by braining p from B to N. 

8. Update L by Subalgorithm 21.3.1. 

9. Setk = k — 1, and go to step 1. 
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10. Compute zy = NTS. 

11. Stopif zy = 0. 

12. Determine index g € argminjen Z;. 
13. Update L by Subalgorithm 21.5.1. 
14. Bring g from N to B. 

15. SetkK=k-+1. 

16. Go to step I. 


Theorem 21.6.1 Under the assumption of nondegeneracy, Algorithm 21.6.1 termi- 
nates either at 


(i) Step 4, detecting lower unboundedness of (21.1), or at 
(ii) Step 11, giving an optimal face together with a pair of primal and dual optimal 
solutions 


Proof Under the assumption of nondegeneracy, the proof of the finiteness of 
Algorithm 21.6.1 is the same as for the simplex algorithm. The meanings of the 
exits come from Lemmas 21.2.1 and 21.4.2. Oo 


In a contraction iteration, the face algorithm solves two triangular systems 
(including one involved in updating L by the subalgorithm 21.3.1), whereas it solves 
only one in an expansion iteration, in contrast to the simplex algorithm that solves 
four. Therefore, the former involves less computational effort than the latter per 
iteration. In addition, there is no need at all for storing the Givens rotations, created 
to update the Cholesky factor. 

The dimension of the end optimal face produced by the algorithm should be more 
than 0, in general, though it would not be necessarily the optimal set. 


21.6.1 Face Phase-I: Single-Artificial- Variable 


Any Phase-I algorithm can be utilized to provide a feasible solution to get the 
face algorithm started. This section introduces a Phase-I algorithm involving only a 
single artificial variable. 

The auxiliary problem used is the same as (14.11), ie., 


min f, 
st. Ax+an4if =b, x, f =O, 


where dy41 = (b — Bh/||b — Bh||. Assume that b — Bh ¥ 0 (see Sect. 14.2). 
At the beginning of Phase-I, set 


B=(l,...,n,.nt+l}, N=@. (21.21) 


Thus, the components of the initial auxiliary feasible solution are all equal to 1, and 
the auxiliary problem can be solved by a slight modification of Algorithm 21.6.1. 
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In step 1, two triangular systems are solved to obtain y. In step 5, n + 1 is taken 
as a candidate for selection of row index p. Once it leaves B (while f becomes 0), 
a feasible solution is obtained. In addition, step 4 should be dropped because the 
auxiliary problem is certainly lower bounded. 

The steps are put in the following algorithm. 


Algorithm 21.6.2 [Face Phase-I: single-artificial-variable] Initial: B = {1,...,n, 
n+l}, N=0,k=n4+1;%=h, f=1; BB! =LL?. This algorithm finds a 
feasible solution to (21.1). 

Same as Algorithm 21.6.1, except for its steps 1-6 replaced by 


. Solve LLTy = n+1- 

. Ifk > m+ 1, compute Ag = ex — BTS. 

. Go to step 10ifk =m-+ 1 or Ag = 0. 

. Determine a and index p such that a = x,)/Ap, = min{x;/Aj; | Aj > 0, j € 
B}. 

5. Ifa ~ 0, update xp = xg — a Ag. 

6. Stop if f = 0 (feasibility achieved). 


BwWN Re 


Example 21.6.1 Solve the following problem by the two-phase face algorithm: 


min f = x; + 2x3 + 3x4, 


s.t. 2x, —x2 —2x3  —Xx4 =-l, 
+2x2 —2x4+x5= 1, 
Xx, —2x2 —x5 = —2, 


eS 0, f= lysed. 


Answer Phase-I: call Algorithm 21.6.2. 
Initial: k = 6, B = {1, 2,3, 4,5, 6}, N =@,x =e, ag = (1,0, ot. B and the 
Cholesky factor of BB! are 


t-1e%et <4 ~1257/379 
B=| 2 22% |. Be =3 
fh ai 2394/1985 5/3 —2804/2109 
Iteration 1: 
¥ = (29/175, —4/35, —36/175)!, 
Ag = (—22/175, —3/175, 58/175, —11/175, —16/175, 146/175)! + 0. 
a = min{1/(58/175), 1/(146/175)} = 175/146, p = 6. 


iy 
2 
4. 
3 

tp = (1,1,1,1,1, 1)! — (175/146)(—22/175, —3/175, 58/175, 


— 11/175, —16/175, 146/175)! 


= (84/73, 149/146, 44/73, 157/146, 81/73, 0)¢. 
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6. x6 = 0, return with 
f = (1, 0, 2, 3, 0)(84/73, 149/146, 44/73, 157/146, 81/73)! = 815/146. 


Phase-II: call Algorithm 21.6.1. 
Initial: k = 6, B = {1,2,3,4,5,6}, N = O, x = (84/73, 149/146, 44/73, 
157/146, 81/73, 815/146)!, B and the Cholesky factor of BB! are 
2-1-2-1 —721/228 
2 —2 1 _ —3 
1 —721/570 5/3 —740/581 
1 2 3 Q =! 721/456 2 905/3458 —392/135 
Iteration 2: 


¥ = (—61/1231, —114/1231, —30/1231, —146/1231), 


Ag = (298/1231, 107/1231, 170/1231, 149/1231, 84/1231, 1085/1231)". 
IAD, 


Yep SS 


a = min{(84/73)/(298/1231), (149/146) /(107/1231)(44/73)/(170/1231), 
(157/146) /(149/1231), (81/73)/(84/1231)(815/146)/(1085/1231)} 
= 1305/299, p = 3. 


XB = (84/73, 149/146, 44/73, 157/146, 81/73, 815/146)! 
— (1305/299)(298/1231, 107/1231, 170/1231, 149/1231, 


84/1231, 1085/1231)! 


= (1211/12867, 109/170, 0, 93/170, 69/85, 15366/8855)_. 


7. B={1,2,4,5, 6}, N = {3}. 


£2158 /881 
dice 3 
—1762/1079 5/3 963/1292 
881/2158 2 —2889/1292 524/387 
9. k= 5. 


Iteration 3: 


= (1, -14/11, -18/11, -6/11)!. 
= ats 1/11, —4/11, 83204/183049)!. 
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5. 
a = min{(1211/12867)/(2/11), (109/170)/(3/11), 
(93/170) /(1/11)(15366/8855) /(83204/183049)} 
= 1203/2324, p=1. 
6. 


Xp = (1211/12867, 109/170, 93/170, 69/85, 15366/8855)! 
— (1203/2324)(2/11, 3/11, 1/11, —4/11, 83204/183049) ! 
= (0, 1/2, 1/2, 1, 3/27. 


7. B= {2,4,5,6}, N = (1, 3}. 


1393/985 
3 
8. L 
1393/985 5/3 —1121/2378 
2378/1121 2 985/1393 —1 
9k=4. 


Iteration 4: 


¥ = (0, -3/2, -3/2, -1)!, 

3.k=4. 

10. zy = N'¥ = (5/2,2)F = 0. 

11. A zero-dimensional optimal face is obtained. The optimal basic solution and 
optimal value are 


x=, 1/2,0, 1/72, 1,372)", «PS 3/2. 


21.6.2 Computational Results 


Preliminary computational experiments were conducted. Compiled using the Visual 
FORTRAN 5.0, test codes were run under a Windows XP system Home Edition 
Version 2002 on an IBM PC with an Intel(R) Pentium(R) processor 1.00GB of 
1.86 GHz memory and about 16 digits of precision. 

The following two dense codes in FORTRAN 77 are compared: 


1. RSA: The revised simplex algorithm, supported by the classical Phase-I (see 
Notation). 

2. FALP: Face Algorithm 21.6.1, supported by Algorithm 21.6.2 being as the Phase- 
I. 
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The test set of problems are the 25 smallest Netlib problems listed in Appendix C 
(Table C.4: AFIRO-SCAGR25). From Appendix E (Table E.4), it is seen that 
average ratios of RSA to FALP are as follows: 

Total iterations ratio:1.51, total time ratio:4.91. 

It is indeed impressive that the face algorithm outperformed the simplex algo- 
rithm with a time ratio near 5. Related computational experiments are reported in 
detail in Appendix E. 


21.7 Affine Face Algorithm 


It is seen from (21.15) and (21.16) that in case xg is degenerate, w would vanish and 
hence leading to a zero stepsize. It is more than that. It is known from (21.17) that 
the difference between the new and old objective values is 


A = 


f — f = —adg. 


Therefore, even though the first k — 1 components of xg are positive in principle, 
the stepsize would still be so small that the decrement of the objective value is 
negligible if some components are too close to zero. Actually, this depends on the 
relative magnitudes of components. To avoid too small stepsizes, the affine interior- 
point algorithm (Part I) exploits affine transformation to attain a relative equilibrium 
between components. In this section, the affine transformation is employed to 
modify the search direction, used in the face algorithm, so that it is not only descent 
but also points to the interior of the feasible region. 

Assume that +g is nondegenerate. Denote by X g the k xk diagonal matrix, whose 
diagonals are components of vector xg, except for the kth diagonal 1, i.e., 


Xp = diag(%1,...,Xx-1, 1). (21.22) 
Consider the affine transformation 
xR= AB 9s (21.23) 


This transformation is invertible and determines a 1-to-1 correspondence between 
Xp and 


w,=(1,...,1, fl. 
k-1 
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Despite the fact that some of the first k — 1 components of xg may be close to zero, 
the corresponding components of x’, are all equal to 1, so that a line search along 
any direction in x/,-space will not lead to a too small stepsize. 

By carrying out transformation (21.23), the subprogram (21.4) is converted to 


min f, 
s.t. (BXg)x', = b, 
xp2=0, n+1¢B. 


The projection matrix onto the null space of BX g is then 
= = alts. 2 
PoT sxe (Bx}B7) BXz. (21.24) 


Therefore, the orthogonal projection of the negative objective gradient onto the null 
space is — Pex, corresponding to vector 


Ap =—XpPex 
in xg-space. Then, combining 
X peg = ex, Bex = —em4i, 
and (21.24) gives that search direction below: 


Ap = —ex + ¥2B"5, (8x37) ne (21.25) 


If the Cholesky factorization Be B! = LL! is available, the y can be obtained 
by solving an upper triangular system only, i.e., 


L153 = -(/vems1, 


where v is the (m + 1)the diagonal of L. 

If x is degenerate, nevertheless, transformation (21.23) is not invertible and 
becomes useless. This difficulty may be overcome as follows. Let 6 be a given small 
positive number. Define 


eae an ee ee (21.26) 


1, If x; <6, 
Then construct an affine transformation with xg in place of xg, i.e., by using 
Xp = diag(%1,...,%x-1, 1), Q1,27) 


instead of (21.22). 
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The difference between the resulting algorithm and the face algorithm lies in 
their search directions only. The overall steps are put into the following algorithm. 


Algorithm 21.7.1 [Affine face algorithm] Given 0 < 6 < 1. Initial: feasible 
solution x. This algorithm solves the reduced problem (21.1). 


1. Settk = n+1,B = A,N = Q, and compute the Cholesky factorization 
BX2,B! = LL", where Xz is defined by (21.27). 
2. Solve LT5 = —(1/v)em+1, where v is the (m + 1)th diagonal of L. 
3. Compute Ag = —ex + x2. BT5, 
4. Go to step 11 if Ag = 0. 
5. Stop if J = {j ¢ B| Aj <0, j An +1} = M Gower unbounded problem). 
6. Determine stepsize a and index p such that a = —Xp)/Ap = minjey —x;/Aj. 
7. Ifa £0, update xp = Xp + aAp. 
8. Update (B, N) by bringing p from B to N. 
9. Update L by Algorithm 21.3.1. 
10. Setk =k —1;ifk > m+ 1, goto step 2. 
11. Compute zy = —NnT5, 
12. Stop if zy > 0 (optimality achieved). 
13. Go to step 1. 


Note This algorithm has two layers of iterations. The inner layer consists of steps 
2-10. 


To reduce the effect of degeneracy in practice, it seems to be favorable to 
replace (21.26) alternatively by 


A x;, if x; >4;, . ; 
| jEB, jAnt+l, 


| 8;, if x; < 8;, 
where 6; > O and n+ 1 ¥ j € B are different small numbers conformable with 
magnitude of the first k — 1 components of xg. 
Example 21.7.1 Solve the following problem by Algorithm 21.7.1, starting from 
feasible solution (0, 0, 3, 2, 4): 


min x; + 5x2 — 2x4. 


S.t. +3x2 +23 —x5=—-l, 
—3x, —x2 +3x4 —2x5 = —2, 
xy —2x2 +2x3 +x5 = 10, 


xj>0, jHl,...,5 
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Answer Take 5 = 10~°. Initial: feasible solution x = (0, 0, 3, 2, 4, =Ayl, 


Iteration 1: 


lk =6, B = {1,2,3,4,5,6}, N = 9. Xp = diag(1, 1,3, 2, 4, yt. Face 
matrix B and the Cholesky factor of BX - pt are, respectively, 
3 1 -1 
—-3-1 3 -2 
BON Sp 

15 -2 -1 

—2449/420 

—5809/1168 3869/419 

769/1121 —4313/1346 —3394/499 

—1402/545 —14233/2934 2753/822 —1941/1519 
2. y = (827/1382, —1205/2826, —269/892, ~2737/4469)!. 
3. Ag = (451/1235, —487/2051, —35/822, —635/2923, —576/763, 

—1732/4469)! x 0. 
5. J = {2,3,4,5} 49%. 
6.a=0, p=2. 
8. B= {1,3,4,5, 6}, N = {2}. 
—5 
—32/5 3019/366 
—2/5 —2682/589 —4040/713 
—1653/505 3573/1456 2273/2022 


9. ES 


10.k=5. 
Iteration 2: 
2. y = (1977/2944, —1095/2176, —73/213, —1517/1917)!. 
. Ap = (80/213, —80/639, 560/1917, —80/639, —400/1917)! #0. 


_J= 3,549. 
. @ = min{—3/(—80/639), —4/(—80/639)} = 1917/80, p = 3. 


WA Mw 


Fp = (0,3, 2,4, —4)! + (1917/80) (80/213, —80/639, 560/1917, 
— 80/639, —400/1917)! = (9, 0, 9, 1, -9)". 


8. B={1,4,5,6}, N = {2,3}. 

4 

—8 —2207/329 

4 1292/2889 2584/2889 
1292/321 —2584/2889 —1 

10, PS: 

11. Zy = (1097/1456, 80/5751)! > 0. 


9 L= 
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12. The zero-dimensional optimal face (optimal basic solution) and optimal value 
are 


%=(,0,0,9,1,-9)), x =-9. 


21.8 Generalized Face Algorithm 


In this section, we generalize Algorithm 21.6.1 to solve the bounded-variable 
reduced problem; i.e., 


min f, 
st. Ax =b, (21.28) 
l<x <u, 


where A and b are the same as in (21.1). The formula for computing the search 
direction A gz and the iteration scheme are also applicable here. As bound constraints 
are added, however, (21.16) for determining the stepsize a and index p has to be 
replaced by 


a =a, =min{a; | je B,j An-+t I}, (21.29) 
where 


(x; = uj)/Aj, If Aj < 0, 
aj=4(4;—-1;)/Aj, If Aj>0, jeB, jAn+1. (21.30) 
loon If A; =0, 


In addition, the optimality should be tested by examining whether the following set 
is empty: 


J ={j ET |Z <O}ULj eM Zz; > 0}, (21.31) 


where I" and IT are defined by (8.9). 
The algorithm can be formulated as follows. 


Algorithm 21.8.1 [Generalized face algorithm] Initial: (B, N), m+1<k <n+l, 
feasible solution x, and the Cholesky factor L of B B!. This algorithm solves the 
bounded-variable reduced problem (21.28). 
Same as Algorithm 21.6.1, except for steps 4, 5, 11, and 12 replaced, respectively, 
by 
4. Compute a; by (21.30). 
5. Determine stepsize a and index p by (21.29). 
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11. Stop if J’, defined by (21.31), is empty (optimality achieved). 
12. Determine index g € arg max jes" |Z;|- 


Example 21.8.1 Solve the following problem by Algorithm 21.8.1: 


min f = x1; —x3+2x4, 


s.t. +3x, +x2 —5x3 +x4 = 3, 
—2x1 +6x3 +x5= 5, 
—6x, —3x2 +3x3 —2x5 = —-11, 
1sx, <9, -2<x <8, 


O0<x3< 12, O<x4< 10, O<x5 < 15. 


Answer There is a feasible solution (1, 2, 1, 3, pr with objective value 6. 


Initial: 
k=6, B={1,2,3,4,5,6}, N=49, 


feasible solution x = (1, 2, 1,3, 1, 6)!, Face matrix B and the Cholesky factor of 
BB! are 


a. f.81 
2 6 1 

B= 
2625.3 =) 
tL =<t% <4 
-~6 

Z 6 2889/1292 


6 2279/637 549/181 
—5/3 —2584/2889 1267/915 —1001/816 
Iteration 1: 
1. v = —1001/816, y = (772/623, 468/623, 27/89, —414/623)". 


2. Ap = (24/89, —205/623, 71/623, 8/89, —90/623, 209/623)! £ 0. 
4. 


1,2,3,4,5 = (C1 — 1)/(24/89), (8 — 2)/(—205/623), (1 — 0)/(71/623), 
(3 — 0)/(8/89), (1 — 10)/(—90/623)). 
5.a=0, p=1. 


6. <p = (1,2, 1,3, 1,6)". 
7. B = {2,3,4,5,6}, N= {I}. 
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—1351/260 
gp | 1351/234 -1057/352 
1351/390 1057/506 —330/139 
~260/193 —843/908 —2128/1401 —251/249 
9.k=5. 


Iteration 2: 


1. v = —251/249, ¥ = (124/63, 220/189, 17/27, —62/63)!. 

2. Ap = (—5/63, —1/63, 0, 2/21, 1/63)! £ 0. 

4. a2,3,5 = ((2 — 8)/(—5/63), 1 — 12)/(-1/63), Ud — 0)/(2/21)). 
5 = 21/2, p= 5: 

6. 


Xp = (1,2, 1,3, 1,6)! + 21/2)(5/63, 1/63, 0, -2/21, —1/63)- 
= (17/6, 7/6, 3, 0, 35/6)". 


7. B= {2,3,4,6}, N ={I, 5}. 
—1351/260 
1351/234 —1762/1079 
1351/390 1079/881 2378/1121 


—260/193 —749/688 1393/985 —1 
9 k=4. 


Iteration 3: 


1. v=—1, y = (124/63, 220/189, 17/27, —62/63)!. 
2.k=4. 
10. Zy = (4/3, 1/6). 
11. J’ =Q. The optimal basic solution and optimal value are 


% = (1, 17/6, 7/6, 3, 0, 35/6)!, % = 35/6. 


21.9 Notes on Face Method 


The quality of an initial partition (B, N) is important to the face algorithm. Although 
initially setting B = A would be a convenient choice, a small B should be favorable, 
ideally containing the “right” (unknown optimal indices) face columns. Therefore, 
it might be advisable to determine an initial B, whose cardinality is no more than 
2m, especially for large-scale problems with n >> m, by using some crash procedure 
based on the most-obtuse-angle heuristic, balanced with sparsity considerations. 
The related computational results with the face algorithm are very well although 
preliminary. This outcome is not surprising because it uses the steepest descent 
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search direction and involves less computational effort per iteration mainly due to 
fewer linear systems involved than the simplex algorithm. 

Based on the dual elimination (Part I), it is possible to design another type of 
algorithm in the following way. By a series of elementary transformations with row 
exchanges, the standard dual problem can be deduced to the following form: 


min -—g=—g+ gry, 


st. Goz=do, z>0, 


(21.32) 


together with Uy + Giz = d,. After the preceding standard problem is converted 
to the reduced form, face Algorithm 21.6.1 or affine face Algorithm 21.7.1 applies. 


Gill and Murray (1973) use the Cholesky factorization BB! = LL, where B is 
the normal basis, to improve the numerical stability of the simplex method. Based on 
the fact that BB! and hence L is independent of the ordering of the columns of B, 
Saunders (1972) proposes an approach to keeping L sparse throughout the solution 
process. It initially carries out row and column permutations to turn the coefficient 
matrix A to the lower triangular form as much as possible and then uses the resulting 
row permutation in subsequent iterations. This approach is clearly applicable in the 
context of the face algorithm. 

It is worth mentioning an interesting result just published by Zhang et al. (2013). 


They present a variant based on updating (BB T) —! by the Sherman—Morrison 
formula to avoid the orthogonal transformations, but it turns out to be unstable (Pan, 
2015). 
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Chapter 22 ® 
Dual Face Method with Cholesky oes 
Factorization 


The same idea of the previous face method applies to the dual problem. The resulting 
so-called dual face method turns out to be even more efficient. In the next sections, 
we put forward the steepest ascent direction, updating of the dual solution, dual 
face contraction, optimality test, and dual face expansion, and then formulate the 
dual face algorithm together with a dual face Phase-I and related computational 
results. The final section introduces the dual face algorithm without the Cholesky 
factorization. 
To begin with, we use notation 


G2) (2) ad 
; 0 1 : ; Ym+1 , Znt+1 , 0)’ 


to transform the standard dual problem into 


max 3 
‘ae 7 (22.1) 
st. Aty+z=c, 220, wm =0, 
where Ae ROMHDXO+TD CEROtD, rank A=m+1,m <n,and 
Aenyi =(—b!,1)7, el Ase. (22.2) 


Assume that this problem is feasible. 
Let (B, N) be a partition of matrix A, where B € RimtDxk and 


rank B =k, 1l<k<m+l. 


So, B is of full column rank. The related variables are, respectively, termed dual 
face and dual nonface variables and matrices termed dual face and dual nonface 
matrices. 
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In the solution process, the kth (last) column of the face matrix is always the 
same as the (n + 1)th (last) column of A. Thus, 


By =(-bT,1)?, el Boel, eli =o. (22.3) 


The partition (B, N) corresponds to dual face 
Dy ={(9,2) ER" XR" | Aly +z=0, zp =0, zw > 0}. (22.4) 


A point on a dual face is termed dual face point. The zero-dimensional face (k = 
m+ 1) has a unique dual face point—vertex. A face is said to be a level dual face if 
the dual objective value is constant over it and said to be dual optimal if the constant 
is equal to the optimal value. 

For simplicity, thereafter (y, zy) € Dy will often be used in place of (y, z) € 
Dn. 


22.1 Steepest Ascent Direction 


We might as well assume that B = {1,...,k — 1,n + 1}. Consider the following 
subprogram: 


max Ym+1; 
st. Bly =cp, (22.5) 


NT y 4+ zy =cn, Zz =0, 


whose feasible region is face Dy. 
Using (m + 1) x (m + 1) projection 


P=1—B(B'B)-'B!, (22.6) 


the objective gradient e,,,1 of the subprogram (22.5) is projected onto the null of 
Bt, yielding the following search direction: 


Ay = Pems1 =@m41 — Bip, B' Big =e, (22.7) 
Azy = —N' Ay. (22.8) 


Proposition 22.1.1 For Ay defined by (22.7), it holds that Bl ay =0. 


Proof By (22.7), it is easy to verify this equation is valid. oO 
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Proposition 22.1.2 The following statements are equivalent: 


(i) Ay #0; (i) (Ay)m41 > 9; (iil) €m+1 ¢ range(B). 


Proof Premultiplying the fist expression of (22.7) by Ay? 


tion 22.1.1 it follows that 


, and then from Proposi- 


0 < Ayl Ay = (Ay)mut, 
which means the equivalence of (i) and (ii). If Ay = 0, then it is known by the fist 
expression of (22.7) that e+; = Bxg, hence e»+ € range(B), so (iii) implies (i). 
Conversely, if e+) € range(B), then there is some vector u such that 
€m+1 = Bu. (22.9) 
Premultiplying the preceding equality by BI, and then from (22.3) it follows that 
ex = B' Bu, 


combining which and the second expression of (22.7) and B of full column rank 
gives u = xg. Thus, (22.9) implies 


€m+1 = Bxg. 


Substituting the preceding equality to the fist expression of (22.7) leads to Ay = 0. 
So (i) implies (iii). Furthermore, it can be asserted that (i) and (iii) are equivalent. 


oOo 
Proposition 22.1.3 [f Ay 4 0, then 
ef Ay = (Ay)m+1 > 9, (22.10) 
ef Ay/||Ayl| > ed, v0 Null(BE 22.11 
m+ AY/IAYIL 2 Cn4i14/llell, Fu € Null(B’ ). (22.11) 


Proof Equality (22.10) directly follows from Proposition 22.1.2. For P defined 
by (22.6), it holds that 


Pu=uf0, VOAuE Null(B!). (22.12) 
It is known from the Cauchy inequality that 


| Pemsilll|Pull = (Pema)! (Pu). 
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Multiplying the preceding inequality by 1/|| Pu|| renders 


|Pem4il] = (Pem41)! (Pu)/|| Pull, 


(Pem+i)! (Pem41)/||Pem4il] = (Pemsi)! (Pu)/|| Pull, 


combining which, P2 =P, pt = P, (22.12), and Ay = Pem+1 leads to (22.11). 
a] 


The preceding proposition implies that Ay 4 0 forms the most acute angle with 
objective gradient e,,+1 of the subprogram, in the null of B" It is therefore termed 
the steepest ascent. In this sense, taking it as a search direction in y-space is the 
“best” choice. 

To compute search direction (Ay, Azy), defined by (22.7) and (22.8), just solve 
the k x k system 


B! Bxp = ex. (22.13) 


Assume that the QR factorization of B is 


Lt 
B=OR =O", |, (22.14) 


where Q « R°"+)x(™+D is orthogonal, and L ce R*** is nonsingular lower 
triangular. Then it is verified that 


paar. 


Thus, the LZ is just the Cholesky factor of B' B. Therefore, the system (22.13) 
becomes 


LL xp = ex, 
whose solution can be obtained by solving the following two triangular systems: 
Lu = ex, Ls xp =U. 


As the solution to the first system is v = (1/j2)ex, where jz is the kth diagonal of L, 
consequently only the following lower triangular system needs to be solved: 


L xp = (1/wer. (22.15) 
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22.2 Updating of Dual Face Solution 


Assume that (y, Zzv) € Dy and Ay ¥ 0. The new dual solution is determined by 
the following line search scheme: 


$=y+BAy, 2y =Zn—BN! Ay, (22.16) 


where the maximum possible stepsize, maintaining Zy > 0, is 
_s T — mids. T T . 
B = 2q/ a, Ay) = min zj/\a; Ay) |a; Ay>0, je Ny; =O. (22.17) 


If some components of Zy vanish, then dual face point (y, zy) is said to be dual 
degenerate, as is a case where stepsize f vanishes, yielding a point the same as the 
old. 


Lemma 22.2.1 Assume that (y, ZN) € Dy with Ay 4 0. 


(i) If NT ay £ 0, then Zy is a boundary point, belonging to Dy. The objective 
value does not decrease and strictly increases if (y, Zn) is nondegenerate. 
(ii) If N? Ay < 0, problem (22.1) is upper unbounded in Dy. 


Proof 


(i) In this case, the new point (¥, Zy) is well-defined. By (¥, Zy) € Dy, (22.17), 
and Proposition 22.1.1, it is known that (j,Zy) € Dy. In addition, it is clear 
that 2q = 0, and hence the new iterate is on the boundary, associated with the 
objective value 


Jm-+1 = Ymoi + B(Ay)m+1- (22.18) 


Furthermore, from 6 > 0 and Proposition 22.1.2, it is known that (Ay)m+1 > 
0, hence Jn+1 > ¥n+1- If (Vy, Zy) is nondegenerate, then B > 0, hence ¥j41 > 
Ym+1, aS indicates that the objective value increases strictly. 

(ii) In this case, it is clear that (¥,Zy) € Dy VB > 0. It is then known by (22.18) 
and Proposition 22.1.2 that the objective value tends to ov, as 6 tends to co. 


oO 


22.3 Dual Face Contraction 


Assume that 2, = 0 and NT ay £ 0. Update (B, N) by bring g from N to B. 
Denoting the new face matrix by B e R¢"+)*x&+D_ we have the following result. 


Proposition 22.3.1 If rank B = k, then rank B =k +1. 
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Proof It is clear that k < rank B < k+ 1. Assume rank B # k-+ 1. Then, there 
is a vector u # O such that ag = Bu. Thus, it is known by Proposition 22.1.1 that 


Bl ay = 0, hence 
ad Ay = ul Bl ay = 0. 
Furthermore, it follows from (22.17) that ala y > 0, as is acontradiction. Therefore 


rank B=k-+1. oO 


According to the preceding proposition, there exists the Cholesky factorization 
Bl sg = iil. 


For the sake of simplicity, it is advisable to insert ag as the second last column 


of Be R™+Dx(K+) (followed by the last column (—b!,1)T). Then, the new 
Cholesky factor can be computed by the following algorithm. 


Algorithm 22.3.1 Assume that the QR factorization of B is given by (22.14). This 
algorithm finds the Cholesky factor L of B'B. 


1. Compute v = Ol ag. 
2. Insert v before the last (k + 1)th column of Rj, yielding matrix H. 
3. Determine Givens rotation Jin41,..., Jk41, where J; is in (k, , )-plane, such that 


;T 
L 
Jpg... Im4i Ht = ( 0 ) 


where L € R&+)*&+1 is lower triangular. 


The validity of Algorithm 22.3.1 can be verified as follows: 
It is clear that 


Jett e+ Jmti1O'B = Jags... Jm4iQ™ lan, ..., 4-1, ag, An+1] 


7T 
L 
= Jeg... ImziH = ( 0 : 


By Proposition 22.3.1, it is known that Le REOxE+ ig nonsingular. Thus, from 
the preceding expression, it follows that 


a 
ee e 3 L ae 
BY B= B OF) 4... dp ten dot 8 = 0,0) ( . = iil, 
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It is seen from step 1 of Algorithm 22.3.1 that it is now required to store the 
Givens rotations in product form, in contrast to the updating of the Cholesky factor 
in the face algorithm. For large-scale problems, therefore, the dual face algorithm 
should be restarted from scratch periodically. 


22.4 Optimality Test 


Assume that Ay vanishes and cannot be used as a search direction, e.g., when k = 
m+ 1. 


Lemma 22.4.1 Assume that Ay is defined by (22.7). If Ay = 0, then Dy is a dual 
level face, and vice versa if there is (y, zn) € Dy such that zy > 0. 


Proof Assume that xg is the solution to B' Bxp = ex. From Ay = 0 and the first 
expression of (22.7), it follows that 


em+1 = Bx. 


Transposing the two sides of the preceding equality and postmultiplying the result 
by y (for any (y, zy) € Dy) gives 


ed y = zhely, 
combining which and Bt y = cg leads to 


<T 
Ym+1 = XpBCB. 


So the objective value is constant over Dy. Therefore, Dy is a level face. 

Now assume that Dy is a level face but Ay ¥ 0. If nT ay < 0, it is known 
by Lemma 22.2.1(ii) that the objective value is upper unbounded over Dy, as 
contradicts that it is a level face; if, otherwise, nT ay £ O, then since (y, Zy) is 
nondegenerate, it is known from Lemma 22.2.1(i) that (9, Zy), defined by (22.16) 
and (22.17), belongs to Dy, with a strictly increased objective value, as contradicts 
that Dy is a level face. Therefore Ay = 0. oO 


Now consider the primal problem of (22.1), i.e., 


min cl x, 


(22.19) 
St. Ax =em41, x>0. 


Theorem 22.4.1 Assume that (¥,Z,) € Dy and that Ay = 0 and xp are 
determined by (22.7). If Xp > 0, then x and (jy, Z) are a pair of primal and dual 
optimal solutions, and Dy is a dual optimal face to (22.1), withxyj = Oand zp = 0. 
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Proof From (22.7) and Ay = 0, it follows that e,.; = Bxg. Also, it is easy to 
verify that x (xy = 0) is a feasible solution to (22.19). On the other hand, it is 
clear that (y, z) (Zp = 0) is a feasible solution to (22.1). The two solutions exhibit 
slackness complementarity. Therefore, they are a pair of primal and dual optimal 
solutions. By Lemma 22.4.1, Dy is a (dual) level face. In fact, it is a (dual) optimal 
face since including an optimal solution. Oo 


22.5 Dual Face Expansion 


If Ay = 0 but xg 7 O, then the level dual face Dy cannot be declared optimal. In 
this case, determine the index 


p € argmin x;, (22.20) 
jeB 
and update (B, N) by 
B=B\{p}, N=NU{p}. (22.21) 


Proposition 22.5.1 The index p determined by (22.20) is different from n + 1. 


Proof It is known by (22.15) that f = (1/7) > 0. As minjeg x; < 0, it is not 
possible to select n + | to be as p. Oo 


Assume that Be RO™+)*(-D is the matrix resulting from dropping the pth 
column of B. Clearly, it is still of full column rank. The associated Cholesky factor 
can be obtained by the following steps. 


Algorithm 22.5.1 Initial: the Cholesky factorization B'B =LLT". This algorithm 
generates the Cholesky factor L of BB! 


1. Drop the pth column from LT, yielding the Hessenberg matrix H. 
2. Determine Givens rotations Jp,..., Jx—1, where Jj is in (i, , +1)-plane, such 


that 
FT 
L 
Jp... JpH = : 
k-1 Pp @ 


where L € R&-)*&—-D is lower triangular. 
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The validity of Algorithm 22.5.1 can be verified simply: since 


Jk-1 ... JpQ™B = Jk-1 saedpQ [aig wees Ap-1, Aptis +++, 4n41] 


7T 
L 
Saeed | ) 


it holds that 


Pa Pa ee ma oe 
BB = Blas... Ferd OMB = CoE )=12 


Seemingly attractive, an alternative way to expand face is to replace (22.21) by 
B= B\{j €B|x; <0}, N=A\B. 


Unfortunately, updating the associated Cholesky factor becomes cumbersome. 


22.6 Dual Face Algorithm 


To be an initial dual face matrix, a submatrix B « R°"™t)** 1 <k <m-+1 with 
small k seem to be preferable, although any of full column rank is eligible. 
The overall steps can be summarized into the following algorithm. 


Algorithm 22.6.1 [Dual face algorithm with Cholesky factorization] Initial: 
B, N, 1 < k < m++1; dual feasible solution (y, Zy) and Cholesky factor L 
of B'B . This algorithm solves the pair of reduced problems (22.1) and (22.19). 


. Solve Ll kp = —(1/)ex for xg, where py is the (m + 1)th diagonal of L. 
. Ifk <m-+1, compute Ay = e741 — Bxg. 

. Goto step 10ifk =m-+1 or Ay =0. 

. Stop if NT Ay <0. 

. Determine index q and stepsize 6 such that 


AB WN Ee 


B=2q/ (a7 Ay) a min {2)/ (a7 Ay) jal ay >O,j¢€ vn] : 


6. If 8 40, update ¥ = ¥ + BAy, Zy =Zy — BNI Ay. 
7. Call Algorithm 22.3.1 to update L. 
8. Update (B, N) by moving p from N to B. 
9. Setk =k + 1, and go to step 1. 
10. Stop if xg > 0. 
11. Determine p € argminjeg Xj. 
12. Call Algorithm 22.5.1 to update L. 
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13. Update B= B\{p}, N= NU {p}. 
14. Settk=k—-1. 
15. Go to step 1. 


Note Although | < k < m + 1 is assumed in the preceding Algorithm, the k = 0 
seems to be favorable. In this case B = 4, N = A, the projection P is the unit 
matrix, and (22.7), which is not well-defined, is replaced by Ay = e,,+41. Then, the 
subsequent steps of the algorithm will go smoothly. 


Theorem 22.6.1 Assume dual nondegeneracy throughout the solution process. 
Algorithm 22.6.1 terminates either at 


(i) Step 4, detecting upper unboundedness of (22.1), or at 
(ii) Step 10, generating a (dual) optimal face together with a pair of primal and 
dual optimal solutions 


Proof Under the assumption of nondegeneracy, the proof of finiteness of the 
algorithm is the same as that for the simplex algorithm. The meanings of its exits 
are derived from Lemma 22.2.1 and Theorem 22.4.1. Oo 


22.6.1 Dual Face Phase-I 


To provide a dual feasible solution to get Algorithm 22.6.1 started, any dual 
Phase-I methods presented in Chap. 15 apply, especially the single-artificial-variable 
algorithm described in Sect. 15.2 fit. In this section, an auxiliary problem, slightly 
simpler than (15.9), is solved by the dual face algorithm. 

Let the pair (y, z) be given with z > 0. If Aly +z =c, then (j, Z) is already a 
dual feasible solution. Now assume it is not the case. 

Define 


h=@—c+Alyy/Z—c+A7 SIL, 
and introduce variable y,,.41. Construct auxiliary problem 


max Yn+1, 


T (22.22) 
st. At ythymy,+z=c, 220, Yynt1 <9. 


It is clear that this problem has a feasible solution (¥, ym-+1, Z), where 


Jme1 = —|Z—c +A 5], 


and is upper bounded, and hence there is an optimal solution. If its optimal value is 
equal to 0, then the part (y, z) of the optimal solution is a feasible solution to the 
original dual problem; else the dual problem is infeasible. 
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In order to decrease the possibility of zero stepsize, it is inadvisable to have many 
zero components of z. A possible way is to set y = 0, z = e, thus 


h = (e-—c)/lle— cll, (22.23) 
and there is a feasible solution 


¥y=0, Ymti=—lle—cl, Z=e, (22.24) 


to the auxiliary problem. Then by setting Al := ve : A], the auxiliary problem is 
converted to a form, similar to (22.1). Consequently, the following Phase-I algorithm 
yields by modifying Algorithm 22.6.1 slightly. 


Algorithm 22.6.2 [Dual face Phase-I algorithm] Initial: B = @, N = A, k = 
0, y = lle —cllem41, Zn = e. This algorithm finds a feasible solution to (22.1). 


1. Set Ay = em +41, and go to step 5. 

. Solve Lb x,y — Bi cag: where jz is the (m + 1)th diagonal of L. 
. Compute Ay = ey,4; — Bxg. 

. Go to step 10 if Ay = 0. 

_ If NT Ay £0, then 


nan &BwWh 


(1) Determine g and f; such that 
p=Z/ (a7 Ay) = min {2)/ (a7 Ay) | al Ay >O jE n} : 


(2) Compute 6 = minf—ym + 1/Aym4i, Bi}; 
else 
(3) Compute B = —Ym41/AYm+1- 


6. Update: } = § + BAy, Zy =Zn — BN Ay. 
7. Stop if ¥m+41 = 0 (dual feasibility achieved). 
8. Update L by Algorithm 22.3.1. 
9. Update (B, N) by bringing g from N to B. 
10. Setk =k + 1, and go to step 2. 
11. Stop if xg > O (dual infeasible problem). 
12. Determine p € argminjeg Xj. 
13. Update L by Algorithm 22.5.1. 
14. Update (B, N) by bringing p from B to N. 
15. Setk=k—-1. 
16. Go to step 2. 
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Example 22.6.1 Solve the following problem by the two-phase dual face algorithm: 


min 4x; — x2 + 2x3, 


s.t. 2x, +x2 +x3 +x4 =. 2, 
Xx] —xX2 —XxX3 —x5=-l, 
2x] —Xx3 tx5= 0, 


xj 20, jgol,...,5. 
Answer The dual problem is 


max 2y — yo, 


s.t.2y) +y2 +293 +21 = 4, 
yl —y2 +z2 = —l, 
yi-y2 —y3t+23= 2, 
y1 tz= 0, 

—y2 +y3+z5= 0, 


zj20, jol,...,5. 


Construct the following auxiliary problem (||c — e|| = 4): 


max yq4, 
s.t. 2y, +y2 +2y3 —(3/4)y4 t+z1 = 4, 
yl —y2 +(1/2)y4 +z2 = —1, 
yi —y2 —y3 —C/4)y4 +73 = 2, 
yt +(1/4)y4 +z4 = 0, 
—y2 +y3+(1/4)y4 +25 = 0 


zj20, j=l,...,5. 


Phase-I: call Algorithm 22.6.2. 
Initial: k = 0, B = O, N = {1,2,3,4, 5}, 7 = (0,0,0, -4)7,z = (1,1,1, 
1 
1,07, 
Iteration 1: 
1. Ay = (0,0,0, 1). 
4. Ay £0. 
5. Nl Ay = (-3/4, 1/2, -1/4, 1/4, 1/4)! £0. 


(1) £1 = min{1/(1/2), 1/./4), 1/0. /4)} = 2, q¢ = 2. 
(2) B = min{4/1, 2} = 2. 


6. ¥ = (0,0, 0, —4)T + 2(0, 0,0, 1)? = (0,0, 0, —2)T. 
ie] (hilt = 2-3 4.1414. tay! = 62,057: 
1/2, 1/2)? 

1. iH 0. 
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8. L = [3/2]. 
9. B= {2}, N ={1,3,4, 5}. 
10. k=1. 


Iteration 2: 

2 tp = (2/9). 

. Ay = (0, 0,0, 1)? — (1, -1, 0, 1/2)1(2/9) = (—2/9, 2/9, 0, 8/9). 
4. Ay £0. 

. NT Ay = (-8/9, —2/3,0, 0)! < 0. 


(3) B = —(—2)/(8/9) = 9/4. 


WwW 


Nn 


¥ = (0,0, 0, —2)? + (9/4)(—2/9, 2/9, 0, 8/9)! = (—1/2, 1/2, 0, 0). 
zw = (5/2, 3/2, 1/2, 1/2)! — (9/4)(—-8/9, —2/3, 0, 0)! 
= (9/2, 3, 1/2, 1/2)". 


7. y4 = 0 achieves the dual feasibility. 
Convert the dual problem to a form of (22.1): 


max y4, 
s.t. 2y) +y2 +2y3 +z= 4, 
yl —y2 +z2 = —l, 

yl —y2 —y3 +73= 2, (22.25) 
YI +z4= 0, 
—y2 +yY3 +75 = 0, 
—2y, +y2 +ya+z6= 0 


zj 20, jol,...,6. 
Phase-II: call Algorithm 22.6.1. 
Initial: k = 2, B = {2,6}, N = {1,3,4,5} y = (1/2, 1/2, 0, =3725", 
zy = (9/2, 3, 1/2, 1/2)! ot v2 ) 
as ’ —(372)/2 /372 
Iteration 3: 


1. ep = 1,273)". 


0 ft 32 1/3 
tape | oh 4 (,\,)- 1/3 
0 0 0 | \2/3 0 
1 0 1 1/3 


3. Ay £0. 
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2 1 20\ /1/3 1 
4, WPAys (21-19) (18) 0 
1 0 00}] oO 1/3 
0-1 10/ \173 ~1/3 


5. B = min{(9/2)/1, 1/2)/(1/3)} = 3/2, q = 4. 
6. ¥ = (-1/2, 1/2, 0, —3/2)? + (3/2)(1/3, 1/3, 0, 1/3)? = ©, 1,0, -D7. 


/2 0 O 
7.L= | J2/2 /2/2 0 
—3//2 —J/2/2 1 
8. B= {2,4,6}, N{1,3, 5}. 
9k =3. 
Iteration 4: 
1, ep tit Aj. 
0 i 0 
0 S10 1 0 
y*10 00 0 0 
1 00 1 0 
3. Ay= 


10. xg > 0. The optimal solution and optimal value to (22.25) are 
¥=(,1,0,-1)', 2=(,0,3,0,1,0)!, 
with the related optimal face 
y=0, yw=1, -3<y3<1, yw=-l. 
Finally, the optimal solution and optimal value to the original problem are 
%=-(0,1,0,1,0)!, f=-1, 
and dual optimal solution is 


¥=(0,1,0)', z=G,0,3,0,)1. 


NI 


22.6.2 Computational Results 


Computational experiments were carried out in the same software and hardware 
environment, as described in Sect. 21.6.2. 
The following two dense codes in FORTRAN 77 are compared: 
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1. RSA: the revised simplex algorithm, supported by the classical Phase-I (see 
Notation). 
2. DFA: dual face Algorithm 22.6.1, supported by Algorithm 22.6.2 being as the 
Phase-I. 
The test set of problems are the 25 smallest Netlib problems listed in 
Appendix C (Table C.4: AFIRO-SCAGR25). From Appendix E (Table E.5), it 
is seen that average ratios of RSA to DFA are as follows: 


Total iteration ratio :1.19, Total time ratio: 10.04. 

It is interesting to compare the performance of FALP (face algorithm) and DFA 
against RSA. It is seen that the iteration ratios of the two codes are 1.51 and 1.19, 
respectively, and time ratios are much higher: 4.91 and 10.04. Impressively, both 
methods outperformed the simplex algorithm with large margins. 

As for a comparison between the face and dual face methods, the latter 
outperformed the former with a time ratio of 2.04. It is not surprising because the 
latter involves systems of order k, generally much less than the order, m + 1, of the 
systems involved in the former. 

The detailed computational experiments are reported in Appendix E. 


22.7 Dual Face Algorithm via Updating (BT B)-! 


This section offers a variant of the dual face algorithm by updating (Bl B)-! 
rather than Cholesky factors. It is originally motivated by the idea of being 
amenable to large-scale sparse problems by getting rid of orthogonal transforma- 
tions. Surprisingly, the variant turns out to be so simple that there is no need 
for solving any linear system although it might be less stable than the dual face 
algorithm. 

Firstly, we consider how to update the inverse in a contracting iteration. 


Lemma 22.7.1 Let Be R°"t+)** with 1 < k < m+ 1 bea face matrix. If 
04ay ¢R"*! satisfies aq ¢ range B, then it is well-defined that 


-1 
n= (ata, —al B(BTB)' BT a,) (22.26) 


Proof Since B is of full column rank k, (BT B)-! e Rk** is well-defined. The 
condition ag ¢ range B implies that the orthogonal projection of a, onto the 
orthogonal complement space of range B does not vanish, i.e., 


dq — B(B' B)"'B")a, = UI — B(B'B)"'B"))ay # 0, 
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and hence it holds that 


ad ag — a) B(BTB)-! Ba, = a) I — B(B'B)"'B a, # 0. 
Thus, (22.26) is well-defined. oO 


Theorem 22.7.1 Let B c RO&"tD** with 1 < k <m+1be a face matrix, and 


let (BT B)-! be available. If B =(B: dq), where ag is determined by step 5 of 
Algorithm 23.1.3, then it holds that 


Gieh (BT B)-! + (BB)! Blajal B(B! B)! —p(BTB)“' Bla, 
—pal B(BTB)! UL 
(22.27) 


where yu is defined by (22.26). 


Proof Proposition 23.1.4 implies that ag ¢ range B. According to Lemma 22.7.1, 
therefore, jz (22.26) is well-defined. Then, since 


ata (8) (Ba;) = B'B Bla, 
al 2 al B ana, 


the validity of (22.27) is easily verified. Oo 


From the preceding theorem, the formula for updating (BT B)-! follows, i.e., 


(BT By! = ( a ) , (22.28) 
ve 


where 
-1 

u= Bl ay, w= (BT B)-!y, w= (aj a, _ uw) >, v=-pu, 

D =(B'B)"!—ww!. (22.29) 

Nevertheless, the column a, should be inserted before the last column of B (see 
step 8 of Algorithm 23.1.3). So, the true new face matrix may be obtained from B 
by interchanging its last two columns. Denote the associated column permutation 
matrix by pr. Then, it follows that 

(BT B)! = (BP)! (BP!) = Pel B)'P?. 


Therefore, the wanted inverse matrix can be obtained by interchanging the last two 
columns and rows of the matrix (22.27), which is referred to as adding permutation. 
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As for how to update the inverse in an expanding iteration, we have the following 
result. 


Theorem 22.7.2 Let Be RO"t)** 1 < k < m+, bea face matrix, and let 
(BT B)-! be of form (22.28). If B is created from B by dropping its last column, 
then it holds that 


(BB)! = D— pov. (22.30) 


Proof Assume that B= (B : dq), and ag is dropped. Then, (22.27) holds. 
Comparing (22.27) and (22.28) gives 


v=—y(B'B) Bla, =D = BB) +y(B'B)'Blagal Bel By), 


from which (22.27) follows. oO 


Therefore, it is quite simple to update the inverse in an expanding iteration: there 
are only (k — 1)*/2 +k — 1 flops involved. Actually, the column selected to leave a 
face matrix, say B, is always among the first k — 1 columns of B. In this case, we 
first move the column to the end position to form a matrix, say B, and then drop it. 
Denote the corresponding column permutation by ol. Then, it holds that 


(BT) = (Bo!) #9!) = oB™B)"'9!. 


Thus, (BT By"! can be obtained by interchanging the last two columns and rows of 
the current inverse (B!B)-! , which is referred to as dropping permutation. 

In practice, an integer vector is enough to record positions of columns to avoid 
moving them all around. The overall steps can be put into the following algorithm. 


Algorithm 22.7.1 [Dual face algorithm: updating (BT B)-} Initial: B, N, 1 < 
k<m-+1, (BT B)-! , dual feasible solution (y, Zy). This algorithm solves the pair 
of reduced problems (23.1) and (23.19). 


. Compute xp = (BT B)~ ley. 

. Ifk <m-+1, compute Ay = e,,4; — Bxg. 

. Goto step Ll ifk =m+1or Ay =0. 

. Stop if NT Ay < 0 (problem (23.1) is upper unbounded). 
. Determine index q and stepsize 6 such that 


AB WN re 


p= (fas) = min[2/(ofa9) Laas = 0.7} 


6. If 8B £0, update y = y+ BAY, ZN = Zy — BN! ay. 
. Update (BT B)-! by (22.28) with adding permutation. 
8. Bring g from N, and insert before the last column of B. 


~ 
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9. Setk = k + 1, and go to step 1. 
10. Stop if xg > 0 (optimality achieved). 
11. Determine p € argminjeg xj. 
12. Update (BT B)-! by (22.30) with dropping permutation 
13. Update B= B\{p}, N= NU {p}. 
14. Settk=k—-1. 
15. Go to step 1. 


It is seen that no linear system has to be solved in the solution process if k = 1 
and B = (an+1) are set initially. 


Example 22.7.1 Solve the following problem by Algorithm 22.7.1: 


min f = 5x, + x2 + 3x4 + 6x6 + 2x7 


s.t. 3x1 — 7x3 + 5x4 — 4x5 — 6x6 — 5x7 =—-—18 
1/3x, + 2x2 + 1/3x3 + —2/3x5+ 25= 0 
—5x2 + 1lx3 — 7x4 — 3x5 + 4x6 — 9 


SO). FS 1;.0597, 


Answer 
The problem is transformed to the following one in tableau form: 


Pal x2 x3 «| x4 | x5 Fi X6 X7 xg RHS 
3 -7| 5 | -4 | -6}| -5 | 18 
1/3 2 1/3 | —2/3 2 
—5 11 | 7 —3 4 -9 
1 1 
5 1 3 6 2 


k= 1, B=ag = (18,0, —9,1)', N = A\B; (B'B)~! = (1/406), 
¥ = 0,0, 0,0)!, Z= 6, 1,0, 3,0, 6, 2, 0)!. 
Iteration 1: 


1. ¥p = (1/406). 
2. Ay = (—9/203, 0, 9/406, 405/406)", 
4.NTAy = (27/203, —45/406, 225/406, —153/406, 45/406, 72/203, 
45/203)! £0. 
5. B = min{0/(225/406), 0/(45/406), 6/(72/203), 2/(45/203)} = 0, g =3. 
dg = (-7, 1/3, 11, 0)!. 
es oe ca 91/7458 ) 
91/7458 517/56043 
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=F 18 3 S: ad. 26255 
Ren ye f 82  -23 2 
id 26 ay a3 24 
1 
9. k =2. 


Iteration 2: 


1. xp = (91/7458, 517/56043)!. 
2. Ay = (—227/2815, —53/13031, —294/5743, 537/542)! £0. 
4. NT Ay = (—642/2639, 314/1267, —84/1873, 781/1631, 1541/5522, 4977/ 
12598). 
5. B = min{1/(314/1267), 0/(781/1631), 6/(1541/5522), 2/(4977/12598)} = 
0,q=5S. 
dq = (—4, -2/3, -3, 0)! 


1027/977 420/271 2223/2947 
7. (BT B)-! =| 420/271 1447/620 827/740 J. 
2223/2947 827/740 1049/1927 
= ae oe: 3 5 -6 —5 
s pu | 13-2, . wee? : 
Tis 23-26 aT 
1 
9. k =3. 


Iteration 3: 


_ xp = (2223/2947, 827/740, 1049/1927)!, 
. Ay = (—125/2599, 733/1485, —134/2941, 878/1927)! <0. 
. NT Ay = (325/16051, 1893/1558, 51/650, 37/348, 1224/997)!, 
. B = min{5/(325/16051), 1/(1893/1558), 3/(51/650), 6/(37/348), 2/(1224/ 
997)} 
= 1/(1893/1558), g = 2. ag = (0,2, —5, 0). 
6. ¥ = (—217/5482, 13765/33883, —1165/31067, 33364/88971) , 
Zy = (299/60, 0, 1591/542, 11689/1977, 1709/1727)!. 
2271/1000 1559/559 684/1115 2371/1581 


ABN 


7. (BT By-1 = | 1559/5359. 2407/670 1217/1953 778/415 
684/1115 1217/1953 1106/3585 298/795 
2371/1581 778/415 298/795 5281/5282 
= 18 5,5: 65 
178231352 1/3 2 
.B= = 
; fi 3 ese5 | os 
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Iteration 4: 


11. The optimal solution and value are 
x = (0, 0.3748, 0, 1.4997, 0, 1.8747, oF >0, f= clits = 0.3748. 
The optimal solution obtained is 
x* = (0, 0.3750, 0, 1.5000, 0, 1.8750, 0), f* = 0.3750. 


Chapter 23 ®) 
Face Method with LU Factorization Ghost for 


The face and dual face methods, introduced in the previous two chapters, involve 
orthogonal transformations. But from a practical point of view, many researchers 
prefer LU factorization. In this chapter, we derive the face method based on LU 
factorization. In the following sections, we derive the descent search direction first 
and then address such topics as the update of the face solution, pivot operation, and 
optimality test. Then we spend the other sections to formulate the face algorithm 
in the tableau form and in the revised form, respectively, before making final 
comments. 
Consider the standard LP problem; i.e., 
T 


min c* x 


(23.1) 
st. Ax=b, x>0, 


where A € R’”*" (m <n),b € R”, c € R"; rank(A) =m. 


23.1 Decent Search Direction 


The quality of the search direction used is crucial to any LP solver. In this section, 
we derive a fresh search direction first. 
The trick is to partition further by 


B=BUB', — rank(B)=™m, (23.2) 


where B € R’"*" is called basis (matrix) and B’ called active matrix. Associated 
items, such as indices, variables, etc., are said similarly. 
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Viewing the objective value f as a variable, we put subprogram (1.10) into the 
system of equations, expressed as 


Xp Xp Xn f| RHS 
B B’ N b (23.3) 
cl ab cl -1 


where entries under xy correspond to the inactive (fixed) part. 
Execute Gauss—Jordan elimination to convert the preceding tableau into the 
following form: 


xp Xp xn f| RAS 


le. BR _ b (23.4) 
el, ef -1| wu 


which is termed (canonical) face tableau. 
Consider the following direction: 


—_ Bla a 
Ax BuB! = ee ) = ( Bee ) : (23.5) 
XB’ CB’ 


Theorem 23.1.1 Let Axgup’ be defined by (23.5). If it is nonzero, the following 
holds: 


Um, B')Axpup’ = 0, (23.6) 
(0, 24) Axpup’ > 0. (23.7) 


Proof The first expression of (23.7) is easily verified by (23.6) and (23.5). 
In addition, it holds by (23.5) that 


(0,2f:) Axwun = epcp > 0, 


which exhibits the validity of the second expression of (23.7). Oo 


The preceding implies that nonzero —Axgup is descent in the null space of 
(Im, B’) and hence is eligible to be taken as search direction. It is more than that. In 
the next section, we will show that it is actually the best, in a certain sense. 
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23.2 Updating of Face Solution 


Assume that xgugy is the current face solution (to subprogram (1.10)) and that a 
nonzero search direction Ax pug defined by (23.5) is available. 
Define the index set 


J={j ¢ BUB’'| Ax; > 0}. (23.8) 


Assume now that J is nonempty, and hence the following line search scheme is 
well-defined: 


X pup! = Xpus — @AXpuB’, (23.9) 
where q@ is the stepsize such that 


a =minx;/Ax; > 0. (23.10) 
jes 


For the new solution Xgyg’, the following is claimed. 


Theorem 23.2.1 Assume that Ax pug is defined by (23.5), and J defined by (23.8) 
is nonempty. If Xgup’ is a feasible solution to subprogram (1.10), then so is XBup’ 
given by (23.9) with (23.10). 


Proof Under the assumption, it holds that 
(BU B’)xXpup = b, X pup’ = 0. (23.11) 
Combining (23.9), (23.6), and the first expression in (23.11) gives 
(BU B’)Sgup = (BU B’)Xgup — a(B U B’)Axpup = b. (23.12) 
Thus, gua’ satisfies the equality constraints. 
Note that x; > OVj € BU B’. Without loss of generality, assume that (B U 
B’)\\I #@. 
It is clear by (23.11) that 
Axj <0, j€(BUB)\J, 
from which and (23.9), for any a > 0, it follows that 
Xj =x; —aAx; > 0, Fe (BUB)\S. (23.13) 
On the other hand, it holds by (23.10) that 


xj/Axj =a =0, jeJ. 
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Multiplying the preceding by Ax; > 0 leads to 
xj; > aAx; > 0, JéeJ, 
which together with (23.13) gives 
Xj =X; —aAx; = 0, jeJ. (23.14) 


Consequently, (23.12), (23.13), and (23.14) together imply that xgug’ is a feasible 
solution to (23.14). oO 


From (23.13) and (23.14), it is seen that only the requirement Xj >0,VijeJ, 
limits the increase of the stepsize, so that a determined by (23.14) is the maximum 
possible stepsize that can be taken along the search direction. Indices in J and 
corresponding variables are often said to be blocking ones. 

Note that k-vector xgug’ is a feasible solution to (23.14) if and only if the 
corresponding n-vector 


is a feasible solution to the original problem (23.1). 
Now, let us explore the effect of the search direction further. 
Assuming Cg 4 0, we introduce the following simple result first. 


Lemma 23.2.1 Fork x (k — m) matrix 


Z= (ue? (23.15) 


Tk-m 


it holds that 


Um, B')Z = 0, and rank (Z) =k—m. (23.16) 


Proof The validity can be easily verified. Oo 


Z is known as a null space matrix of (In, B’ ), which is the coefficient matrix 
of the equality constraints of face tableau (23.4). Thus, the set {—Zu | u € Rem 
includes all possible search direction candidates. If u = Cpv, in particular, it is just 
search direction Ax gupy defined by (23.5). 

For any given u € R*~", the associated update of ¥gug is 


Xpup(u) = Xpup — Zu, 
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where a is stepsize. Premultiplying the preceding by (0, él) and moving terms, we 
obtain the corresponding objective decrement, i.e., 


y(w) = (0, éf) Faun’ — (0,29) fava) =a (0, ef.) Zu, (23.17) 


which not only depends on the direction of u but also depends on the norm of it. It 
is therefore reasonable to restrict u by 


Ill] = lea’). (23.18) 
Using (23.17) and (23.18), we then turn to the following problem: 


max y(u) = a(0, el) Zu 


23.19 
s.t. |]ul] = [lear ||, neRE™, ! 


Theorem 23.2.2 Problem (23.19) has unique optimal solution u* = Cp, corre- 
sponding to maximum objective decrement y (u*) = a||ég’ ||. 


Proof From (23.15), (23.17), and the constraint of (23.19), it follows that 


rw nolo) (8) 


= ae hu 


a [CB ||||u || cos < Cpr, u > 


= al|ég’||* cos < Cp, u>. (23.20) 


It is therefore clear that u* = Cg, is the unique optimal solution to problem (23.19), 
corresponding to y(u*) = a||Egr ||. Oo 


Theorem 23.2.3 ensures that y(u*), being proportional to the square of ||¢g’||, 
is the maximum possible objective decrement that can be achieved. Therefore, the 
search direction —Ax pug’ = —ZCpg’ appears to be the “best.” 

In the case when J is empty, we are then done, as the following says. 


Theorem 23.2.3 Assume that Xgug is a feasible solution and Axpgup: defined 
by (23.5) is nonzero. If J defined by (23.8) is empty, problem (23.1) is lower 
unbounded. 


Proof The emptiness of J implies Axgug’ < 0, which along with xgug’ > 0 leads 
to 


X BUB' = XpuBr — aAxgup => 0, Va>0. (23.21) 
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From the preceding, (23.6), and B U B’xg = J, it follows that 
Bxgp = Bxgp —aBAxg=b, Va>0. 


Therefore, xg is a feasible solution to (1.10) for alla > 0. 
Moreover, by (23.21) and (23.7), it holds that 


clits _ clits = achAxg > 0, Va>0. 


Consequently, the decrement in objective value goes to infinity, as so does a. Oo 


23.3 Pivoting Operation 


ssume > m, Or /). Define the following subset: 
A k B’ #9. Define the following sub 
S = {j ¢ B’| xj; =0, cj = O}, (23.22) 


which consists of all indices of B’ exhibiting complementarity slackness. It is 
obvious that if S is empty, any zero stepsize will never yield from B’. This can 
be ensured at the beginning of each iteration by executing the following tactic. 


Tactic 23.3.1 If S is nonempty, adjust (B’, N, k) by 
B’=B\S,N=NUS,k=k-|S|, (23.23) 


where |S| denotes the cardinality of S (the same below). 


Now assumed that a new face solution xg was computed by (23.9) with a given 
by (23.10). 
Introduce the following index subset: 


Jog ={j € J | xj/Axj =a}, (23.24) 
where J is defined by (23.8). It is clear that 
xy; = 0, Vie Jy. (23.25) 


If Jy C B’ (or BN Jy = B), what to do is just going to the next iteration, where 
Jy will be moved from B’ to N. 
Now consider the other case: Jy ¢ B’ (or BN Jy 4 9). 


Designate by j; the index of the i-th column, e;, of B = Im. Thus, variable x j, 
corresponds to the i-th row of tableau (23.4). 
At first, select a pivot row by the following rule. 


23.3. Pivoting Operation 613 


Rule 23.3.1 [Row Rule] Select the row index p such that 


Jp © arg max Ax;,. (23.26) 
Ji€ Ta , 


Thus, the index j, is determined to leave B. The reason for choosing such a one is 
as follows. 

It is evident that the cosine of the angle between the search direction —Axg and 
the gradient of a constraint x; > 0 equals 


aj =—Ax;/l|Axgl, Vj €B. (23.27) 


By the so-called “heuristic characterization of optimal solution” (pp. 56-60, Pan 
(2014)), for all j; © B, in particular, the constraint x;, = O corresponding to 
the smallest w;,, or the maximum Ax;,, tends to be active at the optimal solution. 
This indicates that the corresponding index should be selected to enter N, and the 
associated row be taken as the pivot row. 

It is noticeable that Ax;, > 0 since jp € Jy € J. Extract the p-th row of B’ 
from tableau (23.4), i.e., 


vi, = el B’. (23.28) 


Then, from (23.5), it follows that 


AXxj, = —e!l Bey, —— > vjcj > 0, (23.29) 


which implies that there exists index j € B’ such that vj¢; < 0. Thus, we are able 
to bring up the following plausible column rule in accordance with Rule 23.3.1. 


Rule 23.3.2 (Column Rule) Select the column index q to enter B such that 
q € argmin{v;c; | j € B’}. (23.30) 


If there is a die, select one corresponding to the maximum Xq. 


Assume now that row index p and column index g have been determined. Update 
the face tableau (23.4) by premultiplying the p-th row by 1/dp,q and adding —dj,q 
times of the row to the i-th row, fori = {1,...,m}, i 4 p. Then, adjust sets 
accordingly by the following tactic. 


Tactic 23.3.2 Adjust (B, B’, N,k) by 


B= (B\{jp}) U(q}, BY = B’\g}, N= NU {jp}, k= k= 1. (23.31) 
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23.4 Optimality Test 


It is noticeable that (23.4) is actually a simplex tableau of the original problem (23.1) 
if one takes into account of both the face and inactive sections. 

Assume that search direction Axg vanishes, and hence line search scheme (23.9) 
becomes meaningless. It is clear that this case occurs if and only if k = m, or else 
Cp =0. 


Lemma 23.4.1 /f active reduced costs Cg vanish, the face Pg defined by (21.3) is 
a level face. 


Proof By cg = 0, it is clear that the equation corresponding to the objective row 
of (23.4) becomes 


Oxp + Oxy +élxy — f= —f. (23.32) 


Since all x € Pp satisfy xy = 0, substituting them to (23.32) results in f = f, 
which means that Pg is a level face. oO 


Since the objective value is constant over the current face, in this case, it is not 
possible to make further progress from the associated feasible solution x, listed in 
the bottom row of tableau (23.4). It is time to test for optimality. 


Theorem 23.4.1 Assume that the active reduced cost Cg: vanishes. Then Pg is an 
optimal face if the following index set is empty: 


T={j EN |G <0}. (23.33) 


Proof Note that cy is nonnegative under the assumption. 

By Lemma 23.4.1, cg’ = O implies that Pg is a level face. Since x belongs to 
Pp, therefore, just examine whether x is optimal. In fact, it is so since Cg, = 0 and 
Cy = 0 together ensure that the simplex tableau (23.4) is optimal. Therefore, Pg is 
an optimal face. oO 


Corollary 23.4.2 Under assumptions of Theorem 23.4.1, the associated x and C 
are a pair of primal and dual optimal solutions to problem (23.1). 


In case when cg, vanishes but T is nonempty, however, optimality cannot be 
claimed. Then perform the following tactic to expand the face, and carry out a new 
round of iterations. 


Tactic 23.4.1 To expand the face, adjust (B’, N, k) by 


B= B'UT, N=N\T, k=k+ITI. (23.34) 
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23.5 Face Algorithm: Tableau Form 


In each iteration, a face tableau in form (23.4) is handled with xgug’ updated 
and (B, B’, N,k) adjusted so that k decreases by one, at least. Such a series of 
iterations referred to as faces contraction are performed until reaching a level face 
to test for optimality (unless detecting lower unboundedness). If optimality cannot 
be declared, we execute Tactic 23.4.1 to expand the face and carry out another face 
contraction. 

We suggest some kind of Phase-1 using “the most-obtuse angle rule” (Sect. 24.3) 
although any simplex Phase-1 applies. However, we will not go into any details here. 

Let B be the initial basis. Alternatively, it might be convenient to take the 
following initial partition: 


B'=A\B,, N=, k=m, (23.35) 
with feasible solution 
Xpus = {XB, XB’}, Xp = 0. (23.36) 


Overall steps are summarized as follows. 


Algorithm 23.5.1 [Face Algorithm: Tableau Form] Jnitial: basis B with parti- 
tion (23.35); canonical face tableau in form (23.4); feasible solution (23.36). This 
algorithm solves problem (23.1). 


. If S defined by (23.22) is nonempty, adjust (B’, N, k) by (23.23). 

. Go to step 13 if Cp = 0. 

. Compute Ax pup by (23.5). 

. Stop if J defined by (23.8) is empty. 

Determine a by (23.10). 

. Ifa > 0, update x pup’ by (23.9). 

Go to step 1 if Jy C B', where Jy is defined by (23.24). 

Determine row index p by Rule 23.3.1. 

Determine column index q by Rule 23.3.2. 

. Multiply the p-th row by 1/ap,q, and add —4j,q times of the p-th row to the i-th 
row foralli =1,...,m+1;iF p. 

. Adjust (B, B', N,k) by (23.31). 

. Ifk > m, goto step 1. 

. Stop if T defined by (23.33) is empty. 

. Adjust (B’, N, k) by (23.34), and go to step 3. 


~~ 
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Theorem 23.5.1 Assume nondegeneracy in a single iteration for each contraction 
process. Algorithm 23.5.1 terminates either at 


(i) Step 4, declaring lower unboundedness, or at 
(ii) Step 13, giving an optimal face together with a pair of primal and dual optimal 
solutions 


Proof It is evident that Algorithm 23.5.1 reaches a level face at the end of 
every contraction process, within finitely many iterations, unless detecting lower 
unboundedness. 

Assume that Algorithm 23.5.1 does not terminate. Then, it must carry out 
infinitely many contraction processes, each of which includes an iteration with 
nonzero stepsize. Therefore, by Theorem 23.2.3, the traversed level faces corre- 
spond to strictly reduced objective values. This contradicts that the number of level 
faces is finite. Therefore, the algorithm terminates. 

By Theorem 23.2.1, exit step 4 indicates lower unboundedness. By Theo- 
rem 23.4.1 and Corollary 23.4.2, exiting from step 13 gives an optimal face together 
with a pair of primal and dual solutions. Oo 


We point out that degeneracy remains a problem, at this stage. But, the 
nondegeneracy assumption in Algorithm 23.5.1 seems to be quite relaxed. Indeed, it 
is unimaginable that a contraction process does not include any iteration involving 
nonzero stepsize. 

It is noted that if stepsize is nonzero, components of Xg, are all strictly positive, 
and hence, any chosen entering X, will never vanish. It is also evident that leaving 
index j, will never reenter B until another faces contraction conducted. 


Example 23.5.1 Solve Beale’s cycling problem by Algorithm 23.5.1: 


min f = —3/4x4 + 20x5 — 1/2x6 + 6x7, 


S.t. x4 +1/4x4 —8x5 —x6 +9x7 = 0, 
x2 +1/2x4 —12x5 —1/2x6 +3x7 = 0, 
x3 +X6 =1, 


xj>0, j=Hl,...,7. 


Initial: x = [0, 0, 1, 0, 0, 0, oF, B= {1,2,3}, B’ = {4,5,6,7}, N=@. 
Face tableau 


xX] x2 x3 X4 x5 X6 X7 
1 0 0 1/4 -8 -l1 9 
0 1 0 1/2 -12 —-1/2 3 
0 0 1 0 0 1 0 
0 0 0 —3/4 20 —1/2 6 
0 0 1 0 * 0 * 
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Iteration 1: 


1. S={5,7} 4, B’ = {4,6}, N = {5,7}; Xeue = U1, 2, 3,4, 6], k=5. 
2. gr =[-3/4,-1/2]! £0. 
1/4 -1 
3. B’ =} 1/2 —-1/2 |, Axgup = [—5/16, 1/8, 1/2, —3/4, -1/2]! #0. 
0 1 
J ={2,3) 4G. 
. a =min{0, 2} =0. 
{2} = Jy £ B’. 
jp = 2, p =2. 
_ min{(1/2)(—3/4), (—1/2)(—1/2)} = -3/8, q =4. 
. Multiply row 2 by 1/(1/2), and add —1/4 and 3/4 times of row 2 to rows 1 and 
4, respectively: 


_ 


xX] x2 X3 x4 x5 X6 x7 
1 -1/2 0 0 —2 -—3/4 15/2 
0 2 0 1 —24 -1 6 
0 0 1 0 0 1 0 
0 3/2 0 0 2 —5/4 21/2 
0 * 1 0 * 0 * 


11. B ={1,4,3}, B’ = {6}, N = (2,5, 7},k=4. 
12. k>m. 


Iteration 2: 


1. S=Q. 

2. 9 = [-3/4, -1/2]' #0. 

3. B’ = [-3/4, -1,1]!, Axgug = [-15/16, —5/4, 5/4, -5/4]! £0. 

4. J=BYAD. 

5. a = min{4/5}. 

6. Xpup’ = (0,0, 1, O]! — (4/5)[—-15/16, —5/4, 5/4, 5/4]! = [3/4, 1,0, 117 
7. B}= Ja CB’. 

8. jp =3, p=3. 

9. min{—5/4}, g =6. 

10. Add 3/4, 1, and 5/4 times of row 3 to rows 1, 2, and 4, respectively: 
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x{ x2 x3 x4 x5 X6 X7 
i =e Sy 0 S8- G@ is 

0 a i it A 6 

0 0 1 0 0 1 0 

0 3/2 5/4 O a ©. B12 
3/4 e * 1 * 1 * 


11. B= {1,4, 6}, B’ = 4, N = {2,3,5, 7}, k = 3. 
12. k=m. 
13. T = Y. Terminated with primal and dual optimal solutions and optimal value: 


x* = [3/4,0,0,1,0,1,0]2, 2* =[0,3/2,5/4,0,2,0,21/2]:, f¢* =—5/4. 


23.6 Face Algorithm 


Algorithm 23.5.1 can be transformed into a practicable version, using original data 
instead of the canonical face tableau as a whole. It involves some key quantities, 
which can be calculated based on (23.10) and (23.11). 


We favor the LU factorization. Assume that B = LU is available. 


(i) Search direction Ax gug’ = {Axg, Axp’} can be calculated as follows. 


At first, solve the following two m x m triangular systems successively for 
= -T 
y=B cp: 


Ulw=cp, Liy=w. (23.37) 


Then compute active reduced costs by 


Ep =cp — By. (23.38) 
Therefore, Ax pz = Cp’ is available. 
To gain Axg = —B~!B’épy, solve the following two m x m triangular systems 
successively: 
Lw = —B'ép,, UAxp = w. (23.39) 


(ii) Assume that pivot row index p is determined. Then the corresponding row of 
B’, ie., ul = el By |B’, can be obtained as follows: 
Firstly, solve the following two m x m triangular systems successively for 


— Rp-T, . 
u=B ep: 


Ulw=e,, Liu=w. (23.40) 
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Then, compute 
vp = Bu. (23.41) 


(iii) Let vg be available. Assume that j, € B was moved to N and q € B’ moved 
to B. 
Then, active reduced costs can be updated for the next iteration by the 
following formula: 


CBr = CB’ + Bue’, B = —Cq/Vq- (23.42) 


Overall steps are summarized in the following algorithm. 


Algorithm 23.6.1 [Face algorithm: revised form] Initial: Given basis B and 
partition (23.35); factorization B = LU; feasible solution (23.36). This algorithm 
solves problem (23.1). 


. Compute Cp by (23.37) and (23.38). 

. If S defined by (23.22) is nonempty, adjust (B', N, k) by (23.23). 
. Go to step 15 if Cp = 0. 

Set Ax pr = Cg, and compute Ax pg by (23.39). 

. Stop if J ={j € BU B'| Ax; > 0} is empty (lower unbounded). 
. Determine a by (23.10). 

. Ifa > 0, update xgup by (23.9). 

. Go to step 2 if Jy C B’, where Jy is defined by (23.24). 

. Determine row index p by Rule 23.3.1. 

10. Compute vp by (23.40) and (23.41). 

11, Determine column index q by Rule 23.3.2. 

12. Update L and U. 

13. Adjust (B, B’, N,k) by (23.31). 

14. Ifk > m, update Cp by (23.42), and go to step 2. 

15. Compute Cy = cn — nT5, where y is defined by (23.37). 

16. Stop if T defined by (23.33) is empty (optimality achieved). 

17. Adjust (B’, N, k) by (23.34), and go to step 4. 


CmONAWNWAWNAN 


Example 23.6.1 Solve the following problem by Algorithm 23.6.1 in two phases: 


min x19 = —6x, + 5x2 — 3x3 — 4x5 + 9x6 — 2x7, 
s.t. 3x, —2x2 +5x3 +4x4 +3x6 +5x7 = 15, 
—x2 +6x3 +2x4 +8x5 —S5x6 4x7 = 18, 
5x, +3x2 —2x3 —8x4 —4x5 x6 —3x7 = 9, 


xj>0, j=l,...,7. 
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Any conventional Phase-1 approach is applicable to provide a basis and a feasible 
solution to get Algorithm 23.6.1 started. Here, we form the auxiliary problem using 
artificial variables. 


Phase-I: 
Introduce artificial variables xg, x9, and x10 to the three equalities, respectively, 
and eliminate them from auxiliary objective function f’ = —xg — x9 — x10 by using 


the equalities. The resulting Phase-I problem is as follows: 


min f° = —8x; — 9x3 4+ 2x4 — 4x5 + 7x6 — 6x7, 


s.t. 3x1 —2x2 +5x3 +44 +3x6 +5x7 +28 = 15, 
—x2 +6x3 +2x4 +8x5 —5x6 4x7 +x9 = 18, 
5x1, +3x2 —2x3 —8x4 —4x5 x6 —3x7 +x1i9 = 9, 


xj>0, j=l,...,10. 


The objective gradient and the feasible solution are, respectively, 


c = [—8, 0, —9, 2, —4, 7, —6, 0, 0, ot and x =[0,0,0,0, 0,0, 0, 15, 18, 9]. 
Initially, we take 
B = {8,9, 10}, B’ = {1,2,3,4,5,6,7}, N=9, 


so that B and its LU factors are all unit matrices, and xz, = 0. 
Iteration 1: 


1. 930, ops = 80-99, 4,961", 
2. S={2,4,6}4 9, B’ = {1,3,5, 7}, N = (2,4, 6},k =7. 
3. ép, £0. 


4. L=U=h, w= —B’cp = —- 


no w 


5 0 5 
6 8 4\|[-8,-9,-4,-6]T 
=e 


= [99, 110,12]! , Axpupr = [99; 110, —12,—8, —9, =4, 6]! 4 0. 
5. J={8, LD. 
6. w = min{5/33, 9/55} = 5/33. 
7. Xp = [15, 18,9, 0, 0,0, 0]! — (5/33)[99, 110, —12, -8, -9, —4, —6]! 
= [0, 4/3, 119/11, 40/33, 15/11, 20/33, 10/11]!. 
8. {8} = Ja CB’. 
9. jp =8, pH. 
10. u=w =e), vp = [3,5,0, 5]. 
11. min{3(—8), 5(—9), 5(—6)} = 5(—9), g =3. 
5/6 10 6 10 

12. Permuted L = 1 00),U =] 0-—5/60 

“1/8975 1 0 O11 
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13. B = {3,9, 10}, B’ = {1,5, 7}, N = (2,4, 6, 8}, kK =6. 
14. k=6>™m, Ep = [-8, —4, -6]!+(—(—9)/5)[3, 0, 5]? = [-13/5, —4, 3]! 


Iteration 2: 


2. S=9. 
3. CR, #0. 


3.0 5 —13/5 —36/5 20 
4. —B’cp =-|0 8 4 —4|= 20 |,w = | —358/15 
5 —4 —3 3 6 78/25 


Axpup’ = [—36/25, 716/25, 78/25, 13/5, —4, 3]! 4 0. 

_ J ={9,10, VS. 

_ @ = min{25/537, 1328/383, 10/33} = 25/537. 

7. gp = [15/11,4/3, 119/11, 40/33, 20/33, 10/11]! — (25/537)[-36/25, 
716/25, 78/25, —13/5, —4, 3]? 

= [1166/815, 0, 2839/266, 2625/1969, 1560/1969, 911/1184]!, 

8. (9} = Ja ¢ BY. 

9. jp =9, p=2. 

10.e. = [0,1,0)',w = [0,-6/5,0]!,u = [-6/5,1,0]!, vy = 
[—18/5, 8, 2]. 

11. min{—13/5, —4, 3} = —4, J) = (5},q =5. 


5/6 10 6 80 
12. Permuted L = 1 00);,U=j] 0-20/30}. 


=1 731/51 0 01 
13. B = (3,5, 10}, B’ = {1,7}, N = {2,4, 6, 8,9}, k=5. 
14. k > m, ép = [—13/5, 3]! + (—(—4)/8)[—18/5, —2]? = [—22/5, 217. 


nN 


Iteration 3: 


2.S=9%. 
3. Ep, £0. 
3.5 16/5 —8 
fGen s|ayp 4 fea =| —8|,w=| assis |, 
5-3 28 584/25 


Axpup’ = [16/25, —37/25, 584/25, —22/5, 2]! £0. 
_ J = (3,10, 40. 

. @ = min{959/429, 5514/1125, 911/2368} = 911/2368, Jy = {7}. 
7. Xp = [1166/815, 1560/1969, 2839/266, 2625/1969, 9111/1184]! 
—(911/2368)[ 16/25, —37/25, 584/25, —22/5, 2]! 
= [1509/1274, 625/459, 3249/1927, 2687/888, oF. 

8. Jy C BY. 


nN 
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Iteration 4: 


2. S={7}, B = (3,5, 10}; B2= {1}, N = {2,4,6,7,8,9},k=4>m. 

a. Gn £0, 

4. —B'éy = —[3, 0, 5]![—22/5] = [66/5, 0, 22], w = [0, 66/5, 484/25]. 
Axpup’ = [66/25, —99/50, 484/25, —22/5]! + 0. 

5. J = (3, 10} £0. 

_ @ = min{485/1081, 143/1642} = 143/1642, Jy = {10}. 

7. Xp = [1509/1274, 625/459, 3249/1927, 2687/888, |]! — (143/1642)[66/25, 
~99/50, 484/25, —22/5]! 

= [24191/25343, 135/88, 0, 75/22]!. 

8. {10} = Jy ¢ BY. 

9. jp = 10, p =3. 

10. e3 = [0,0, 1]!, w =e3, wu =[-1/5, 1/2, 1]!, v = [22/5]. 

11. min{—22/5}, Jy = {1},q = 1. 


lon 


5/6 10 6 8 60 
12. Permuted L = 1 00],U =| 0 —-20/3 3 
—1/3 1/51 0 0 22/5 


13. B = {3,5, 1}, B’ =, N = {2,4, 6,7, 8,9, 10}, k = 3. 
14. k =m. Phase-I ended while all artificial variables left the basis. 


Phase-II: 
The preceding Phase-1 produces an initial feasible solution 


¥ = [75/22, 0, 24191/25343, 0, 135/88, 0, o]!, 


and the related basis B = {3, 5, 1} with (permuted) LU factors. 

Referring to the original objective gradient c = [—6, 5, —3, 0, —4, 9, ait and 
setting B’ = A\B = {2,4, 6,7}, N = 9, we get Algorithm 23.6.1 started as Phase- 
Il as follows: 

Iteration 5: 


l.cg = [-3, —4, -6]!, w = [—1/2,0, -15/11]!, ¥ = [3/11, -13/11, 


—15/11]!; 
| : ae oa 3/11 a 
cy = a eae ~13/11 | = 
| 2 ae 15/11 ale 
=) 30/11 
2. S= {2,6} 4G, B’ = {4,7}, N = {2, 6}. 
3. bp 20, 
4 5 574/11 332/11 
7 —106/11 
4, Bey = S| 2 A ad =| 332/11 |, w= 892/33 
8 —3 —938/11 —4434/55 


Axpup’ = (2593/121, —3038/247, —2217/121, —106/11, —30/11]! ¥ 0. 


23% 


nN 


13 
14 
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SHB 4G. 
_ a = min{191/4288}. 
. ¥g = [24191/25343, 135/88, 75/22, 0, 0]! — (191/4288) 
[2593/121, —3038/247, —2217/121, —106/11, —30/11]! 
= [0, 2007/964, 7204/1705, 743/1731, 233/1918]!. 
. B}= Jo £ BY. 
. jp =3, pH. 
_w = [1/6,1/5,-3/22]', u = [5/22, -3/44, -3/22]', vg, = [41/22, 
1as1iyt. 
. min{—106/11, —30/11} = —106/11, J; = {4}, g =4. 


—1/2 -2/71 -8-4 5 
. Permuted L = | —1/4 10],U= O 7 5/4}. 
1 00 0 041/7 


. B= {4,5, 1}, BY ={7}, N = (2,3, 6}, k= 4. 
_k>m, ég =[-30/11]! + (—(— 106/11) /(41/22)) [14/11] = [158/41]. 


Iteration 6: 


2 
3. 
4. 


fon 


8. 


S=G. 

CB x 0. 

—B'ép = —[5, 4, —3]"[158/41] = [-790/41, —632/41, 474/41]!, 

w = [474/41, —1027/82, —4898/287], 

Axp = [—729/277, —2133/1681, —2905/997, 158/41]! + 0. 

SHIH AG. 

. @ = min{335/10627}. 

. Xp = [743/1731, 2007/964, 7204/1705, 233/1918]! — (335/10627) 
[—729/277, —2133/1681, —2905/997, 158/41]! = [12244/23905, 

87/41, 177/41, oy. 

{7} = Jy CB’. 


Iteration 7: 


2s 
3. 
15. 


S={7},B ={4,5, 1}; B2=9, N = (2,3,6,7},k=3=m. 


cp = [0, —4, —6)!, w = (0, -4/7, ~37/41]!, y = [-37/41, —34/41, 


(2) esa apa ee 

CN = —}-1 6-5 4 —34/41 | = = 0. 
| ; 3-2 1-3 27/41 ne 
—2 158/41 
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16. T =. Terminated with primal and dual optimal solutions: 
x* = [177/41, 0, 0, 12244/23905, 87/41, 0, oF, 
y* = [-37/41, —34/41, 27/41)", zk = [0, 178/41, 212/41, 0,0, 
115/41, 158/41]!. 
Optimal value: f* = —1410/41. 


23.7 Notes on Face Method with LU Factorization 


The face algorithm has the following features: 

At first, unlike the simplex algorithm, which uses the search direction along some 
descent edge of the underlying polyhedron, the new algorithm uses a combination 
of all edges corresponding to nonzero active reduced costs. 

Second, as a result, the new algorithm no longer moves from vertex to vertex, 
but from face to face, with a sequence of corresponding face points, until reaching 
an optimal face together with an optimal point. Therefore, it would have more 
opportunities to bypass degenerate vertices. 

Third, the new algorithm is stable, compared with simplex algorithms, since its 
pivot is selectable, to some extent, while the latter can be arbitrarily close to zero. As 
we all know, the latter sometimes fails to maintain a usable basis and has to restart 
from scratch. 

As for the computational cost in a single iteration, the new algorithm is about the 
same as the simplex algorithm. It solves four triangular systems, just as the latter. 
Using LU factorization, it can be implemented as in the traditional simplex context. 

Finally, the unique and promising feature of the new algorithm might be the 
maximum objective decrement a||Gg’||? achieved per iteration. We expect that 
the number of iterations required could be reduced significantly. At this stage, 
however, we cannot talk much about its actual performance. Among the search 
directions (3.30) and (24.37), e.g., it is unknown which one is better, although we 
used the former. Alternatively, instead of the whole Cg’, it is possible to use a part 
of it to formulate the search direction, e.g., such that 


I@jl=y, fe, (23.43) 
where y is a threshold decreasing in the solution process. All of these need 


to be considered. The implementation of the new algorithm and computational 
experiments are expected. 


Reference 
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Chapter 24 ® 
Dual Face Method with LU Factorization Ghost for 


It is fascinating to develop the dual face algorithm based on LU factorization. We 
handle this topic in this chapter. 

In the next section, we first put forward the key to this method, which is important 
for understanding the subsequent derivation. In the following sections, we discuss 
topics, such as the ascent search direction, update of the dual face solution, pivot 
operation, and optimality test. We spend the subsequent two sections to formulate 
the dual face algorithm in the tableau form and revised form, respectively, and make 
remarks on the algorithm in the final section. 

First of all, we introduce the so-called pseudo-basis, defined as the submatrix 
consisting of k independent columns of A (Pan 2005). If the k is less than m and the 
column space includes J, it is a deficient basis, and if k = m, it is a normal basis. 

Now turn to the dual problem of (23.1), i.e., 


max bt y 


(24.1) 
S.t. Alytz=c, z>0. 


Partition A to (B, N), where B is m x k pseudo-basis with 0 < k < m. 


Definition 24.0.1 A dual face is a dual feasible point/solution set, defined as 


Dy ={(y, zw) € R™ x R" | Bly = cB, NT y +2N =cn, 7B =0, zw = O}. 
(24.2) 


The N and B are termed dual face matrix and dual nonface matrix, respectively. As 
zp = Ois fixed, the B is termed inactive. 

The dual face Dy is an (m — k)-dimensional set. In particular, if k = m, Dy is 
a zero-dimensional dual vertex, and if k = 0, it is the m-dimensional dual feasible 
region itself. Usually, 0 < k < m is assumed. Any solution in Dy is called dual 
face point or solution. 
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Definition 24.0.2 The dual face is a dual level face if the dual objective value is 
constant over it. A dual level face is optimal if the constant equals the dual optimal 
value. 


24.1 Key of Method 


Consider the following least squares problem: 


min |b — Bxal. (24.3) 


xBER’ 
The unique solution to the preceding is known as 
<p — (B' B)~'B"p. (24.4) 
The residual of (24.3) at solution xg is a key quantity, that is, 
Ay = b— Bxg. (24.5) 


It is clear that the Ay vanishes if and only if B is a deficient basis. Otherwise, it 
can be verified that 


Blay=0,  blay>o, (24.6) 


which means that the nonzero Ay is in the null of BI, and is an ascent direction of 
the dual objective function. In fact, Ay is just the orthogonal projection of b onto 
the null of B!. As the closest to the dual objective gradient b in the null of Br, it is 
a favorable search direction in the y-space. 

Without loss of generality, assume that B consists of the first k columns and R 
of the first k rows such that the k x k submatrix Br has full rank. 

Using 


N=A\B,  R.={l,...,m}\R, 


we put problem (23.1) into the following tableau: 


(24.7) 
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Carrying out the Gauss—Jordan elimination to the preceding, we convert Br into 
unit matrix, and eliminate all entries below it, yielding the following tableau: 


xp xy  f | RHS 


I, Nr br 
nee he (24.8) 
ee al || ee 


where Cy is assumed nonnegative. 
Tableau (24.8) corresponds to the following dual subproblem: 


max bl yp + bF yr, 


s.t. —p + = (2 5 = 0, = 0. 
(it at] G ZN CN is 


For the preceding, the counterpart of least squares problem (24.3) is much simpler, 


that is, 
- 2 
: br Tk ) 

min _ x P 24.10 
XB eR‘ ( br, ) ( 0 ‘ ¢ 

which exhibits its unique solution xz = b R with residual 
ay=(_. y (24.11) 

br. 


If b R. # 0, Ay is eligible to be a search direction in y-space. The matching search 
direction in z-space is then 


Azgp=0, Azy =—Np dp. (24.12) 
Tableau (24.7) can also be handled by block Gauss—Jordan elimination. Premul- 


tiplying the first row by Be’ , adding — Br. times of the first row to the second row, 


and adding el times of the first row to the bottom row, we convert (24.7) to the 
following tableau: 


XB XN f RHS 
I BR'Nr Bribe 
Nr. — Br. Bp Nr br. — Br. Bp br 


cl —clB,'Ne -1| —clB;z'be 


(24.13) 
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Since the preceding is equivalent to (24.8), their corresponding entries are equal, 
e.g., 


éy =cy—NABz' ce, Nr. = Nr, — Br. By! Nr. (24.14) 


From the equality constraint of (24.9), it is seen that yr = 0 is actually fixed 
since so is zg = 0. Dropping such an inactive part, subproblem (24.9) becomes 


max by YR. 


: (24.15) 
s.t. Nr yr +ZN =n, zn = 0. 


Designating the feasible region of the preceding by Diy, we have the following 
result. 


Lemma 24.1.1 /f {yr.,zv > O} is in Diy, then {zg = 0, yr = By! (cp - 


BE yr.), YR.» ZN = O} is in Dy, defined by (24.2), and vice versa. 
Proof 


CN =cy — NEBR" cp, Nr. = Nr, — Br, Ba' Nr. (24.16) 


Assume that the former point is in Dy: From definition of Dy and (24.16), it 
follows that 


zy = ey — NE yr, (24.17) 
= ey — Nf ages — (Wf — NP ag" BE) ve 
= cv — NE yr, — N} By? (ce - BR.YR.) 
Substituting 
yr=B, (co - BR.YRe) (24.18) 
into the preceding leads to 
Nh yr + Ni yr, +zN =cn, 


which implies that {y = of, yes ZN} Satisfies 


nly +2ZnN =cn. 
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In addition, (24.18) also implies that 


T 
Bly = (BE BR.) (ve ye.) = Cp. 


Therefore, the latter point is in Dy. 
Conversely, assume that the latter point is in Dy. From the definition of Ny 
and (24.18), it follows that 


ZN = CN — nly 
= cy — Ni B," (ca = BR.YR.) = Ni yr. 
= ey — WE BRT cn — (WE. - NE BGT BE) yi 
which along with (24.16) leads to 
NE yr, + zy =¢y. 


Therefore, the former point is in D4, and the proof is complete. oO 


Lemma 24.1.1 establishes a 1-to-1 correspondence between points of Dy and 
Dy. In particular, it is obvious that {yr. = 0, zw = Cn = Of € Dy corresponds to 


[yr = Bz" cp, yr, = 0, zp = 0, ew = En o| € Dy. (24.19) 
Thus, solving subproblem (24.15) amounts to maximizing the objective of (1.7) over 
Dn. 


Theorem 24.1.1 /f B is a deficient basis, then Dy is a dual level face, as well as 
D\y. 
N 


Proof Without loss of generality, assume that k < m. 
Under the assumption, the residual Ay given by (24.11) is zero, and hence so is 
b R,- Thus, Dy is a dual level face, over which the objective of (24.15) equals zero. 
By Lemma 24.1.1, any {yr., zw = O} € Di, corresponds to 


{ze =a ng (ce = BR.YRe) > VRes ZN = o| € Dn. 
Therefore, the objective value of the original dual problem (24.1) at the latter 
solution equals 

bl yet bh yr. = bE BR : (ca = BR.YR.) oP bh yr, (24.20) 


= bE BR" cp — yp Br. By be + ybbR 


= bE Be" cp + a (br. _ Br, By'br) 7 
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On the other hand, a comparison between the right-hand sides of (24.8) and (24.13) 
gives that 


Combining the preceding and (24.20) leads to 


bh yr + bE yr, = bE BR "cB, 


which is constant over Dy, and the proof is complete. oO 


Corollary 24.1.2 If br, in tableau (24.8) vanishes, then the corresponding Dy is 
a dual level face. 


Nevertheless, it turned out that this approach coincides with one already used 
in Pan (2004, 2005). So, it appears to be blocked. Fortunately, the barrier can be 
broken by further partitions. 


24.2 Ascent Search Direction 


In this section, we derive a new dual search direction through a so-called dual face 
subprogram. 

The trick is to partition further. Let N’ be subset of N and R; subset of R,, and 
set 


N=N\N, Re=RA\R IN| =I =. (24.21) 


In particular, assume NR = 0, Nr, = In, and Nr. = 0, so that tableau (24.8) can 
be written 


XB XN’ XN f RHS 
Tk Nr br 


Ty Nr, br, (24.22) 
Nr. br. 
et, et -1 


which corresponds to dual face with zg = 0 fixed. 
Handled from the dual side, the preceding is termed (canonical) dual face 
tableau, although it still represents the problem (23.1) from the primal side. 
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Using the active part of (24.22), construct the so-called dual face subprogram 
below: 


max be yR, + bY yr, 


Ty YR ) () (*) (24.23) 
s.t. —p = + =(— }, zy,zy = 0. 
& a) ee ZN CN sia 


We will use the preceding to create a search direction to take one step forward. 
Consider the following direction: 
AYR,URe = PRURe» (24.24) 


Azwun = (—br,, —Nh bp, — N} bp}. (24.25) 


Then, the following can be claimed. 


Lemma 24.2.1 Jf Ayr,ur, is nonzero, the direction defined by (24.24)-(24.25) is 
ascent in the null of the coefficient matrix of constraint equations, with respect to 
subprogram (24.23). 


Proof It is ascent, since b R,UR, is the gradient of the objective function. In addition, 
the following equation is easily verified: 


oe os = eet 24.26 
& at) Ge Azw 0 i 


Therefore, the claim is valid. Oo 


Lemma 24.2.1 ensures that the direction defined by (24.24)—-(24.25) is eligible to 
be a search direction. In fact, {b Ry b R,} is the steepest ascent direction in {ypr,, yR,}- 
space. 

In particular, we prefer matrix BU N’ being deficient basis, because b R, Vanishes 
in this case, simplifying formulas (24.24)—(24.25) as 


Ayriur. = {br,.0},  Azwun = {-br,, —NE dR}. (24.27) 


Furthermore, from a numerical point of view, it seems reasonable to use 
normalized br, instead. This leads to the following search direction: 


Ayrur. = {be/lbell.0}, — Azwun ={—Ayas,—Neg Aver}. (24.28) 
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24.3 Updating of Dual Face Solution 


Assume that search direction Azyyy was determined by (24.25). 
Introduce the index set 


J={je NUN | Az; <0}. (24.29) 
If J is nonempty, the current dual feasible solution to (24.23), 1e., 
YR\UR, = 9, ZNWUN = Cn’uN = 9, (24.30) 
can be updated by the following formula: 
SRR. = BbRuR,» (24.31) 
énvun = Cnrun + BAzNiun, (24.32) 
where f is the maximum possible stepsize that can be achieved, i.e., 


p= min —é;/Azj > 0. (24.33) 


Dual feasible solution is said to be degenerate if some component of c,; is 0. In 
this case, dual stepsize 6 vanishes, and the “new” solution is not really new, but the 
same as the old. 

Using the preceding notation, we claim the following result. 


Theorem 24.3.1 If J defined by (24.29) is nonempty, then (24.31)-(24.32) along 
with (24.33) give a new dual feasible solution to subprogram (24.23). 


Proof Note that f is well-defined by (24.33) under the nonempty assumption on J. 

Since old solution (24.30) satisfies the equality constraint of (24.23) and direction 
(24.24)-(24.25) satisfies (24.26), the new solution satisfies the equality constraint. 

From (24.32)—(24.33), it follows that 

B < —c;/Az;, jeJ. (24.34) 
Besides, (24.29) ensures that 
Azj <9, jeJ. 

Thus, multiplying (24.34) by Az; gives 
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which along with (24.32) leads to 
€; = 0; + PAZ; = 0, jeJ. (24.35) 
In case of (N’ U N)\J 4 9, on the other hand, (24.29) implies that 
Azj>0, je(NUN)\J, 
combining which, Cyuny => 0, and 6 => 0 results in 
cj =¢j + BAz; = 0, JE (NUN)\J. (24.36) 


Therefore, it holds that Cyuy > 0, and the proof is complete. oO 


Theorem 24.3.2 If search direction is determined by (24.24)-(24.25), the new 
solution (24.3 1)—-(24.32) increases the dual objective value by 


Af = B(lbr, I? + lle), (24.37) 


with respect to dual subprogram (24.23). 


Proof By (24.31), the old zero-valued solution in yr,ur. space is updated to the 
new solution Bbr,ur,, and hence the corresponding objective value increases from 


zero up to BoE op bRUR. which is equal to (24.37). oO 


If the search direction (24.27) is used instead, the corresponding objective 
increment becomes £ IlbR, ||, whereas the increment becomes {|b R, || if the direc- 
tion (24.28) is used. 

In practice, it is not necessary to update the dual solution in y-space iteration by 
iteration, since the final one can be computed cheaply at the end of the solution 
process (this will be clear a little later). It is also possible to update Cyuy by 
using the dual face tableau. It is noticeable, for instance, that using formulas (24.27) 
and (24.32) amounts to adding —Bbr, times of rows corresponding to R, to the 
bottom row. 

As for J being empty, a case when stepsize 6 is not well-defined, we are still 
done as the following says. 


Theorem 24.3.3 Assume that briUR. is nonzero. If J = %, the problem (23.1) is 
infeasible. 


Proof From the second half of the proof of Theorem 24.3.1, in case of J = 
%, new solutions (24.31)-(24.32) are dual feasible to (24.23) for all B > 0. 
Under the condition b R\UR, ¥ 9, the corresponding objective increment, given by 
Theorem 24.3.2, goes to oo, as so is B. 

Therefore, dual subprogram (24.23) is upper unbounded, and hence there 
exists no feasible solution to the corresponding primal subprogram, as well as to 
problem (23.1). oO 
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24.4 Pivoting Operation 


In this section, we discuss pivoting operations and associated index set adjustment. 
Assume that new dual solution Cyyuy was determined by (24.32) along 
with (24.33). Introduce the index set 


J'= ett —c;/Azj, (24.38) 


where J is defined by (24.29). It is clear that for each j € J’, component ¢; reaches 
the critical point zero. Related variables or indices are said to be blocking ones. 
There are two types of iterations as follows: 


(A) 72 N’/ J' is nonempty (or J’ ¢ N). 


There is then no need for pivoting, and the involved computational cost is quite 
cheap. Before going to the next iteration, what to do is only to execute the following 
tactic. 


Tactic 24.4.1 Assume that row index set [ corresponds to J. Update {N’, B, Ri, 
R, k} by 


N=N\J,B=BUJ, 


fs - 2 24.39 
Ri =Ri\I, R= RVI, kK=k+ JI. ( ) 
(B) J SP NTN Sis empty (or J’ Cc N). 
There is a need for pivoting. To do so, execute the following rule first. 
Rule 24.4.1. [Column rule] Select the column index q such that 
q € arg min Az;. (24.40) 
jes 
Then determine row index p such that 
p € arg max |@jq|. (24.41) 
iER. 


There will be the following two cases arising: 


G) a4pqg #0. 
We update the tableau by using dp, as the pivot (see below) and then update 
the index sets conformably by the following tactic. 


Tactic 24.4.2 Update {N, B, Ro, R, k} by 


N=N\q, B=BUgq, 


(24.42) 
Roe =Re-\p, R= RUp, kK=K+1. 
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Gi) apg = 0. 
There are multiple choices for row pivoting. We bring up the following plausible 
one. 


Rule 24.4.2 [Row rule] Assume that column index q was determined by (24.40). 
Reselect row index p such that 


p € arg min{—b;G;,q | i € Ri}, (24.43) 


corresponding to jy € N’. 
Using the preceding notation, we have the following result. 
Proposition 24.4.1 Az, and —D pain, gq have the same minus sign. 


Proof It is evident that aj, = 0 implies N R.,q = O, and hence 
asi T — -_ 
(Nr..q) br. = 9. (24.44) 


In addition, by (24.29) and q € J, it is known that Az, < 0. Hence, combin- 
ing (24.25) and (24.44) leads to 


Azg = -Ni br, = \ > -bidig < 0. 


iER, 


The preceding along with (24.43) gives —b p@p,q < 9, which has a minus sign as 
Azq. Oo 


The preceding ensures that the pivot dp, is nonzero. It is more than that. It 
also implies that Rule 24.4.2 conforms to the heuristic characteristic of the optimal 
solution, in the dual context. 

Once the pivot is determined, what to do next is to update the tableau. To do so, 
modify the first m rows by premultiplying the p-th row by 1/dp,4 and adding —dj,q 
times of the row to the i-th row, fori € Rj UR., iF p. 

Then, update the index sets by the following tactic respectively. 


Tactic 24.4.3 Update {N, B, N’, Ri, R, k} by 


N=N\q, B=BUgq, N=N\ip}, (24.45) 
Ri} =Ri\p, R= RUp,N=NU{jpj, kKa=K+1. , 
We ignore related column and row permutations in conjunction with the tableau. 
After all, it is not necessary to move around entire column or row of entries in the 
implementation. 
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24.5 Optimality Test 


Let (24.22) be current dual face tableau. It is noted that if k = m, none of 
formulas (24.25), (24.27) and (24.28) is well-defined, and if k < m but brwR, = 0, 
all these directions vanish. 

Consider the following solution given by tableau (24.22): 


Xp = bp, Xwun = 0. (24.46) 
Clearly, if B is not a deficient basis, such a solution does not satisfy the equality 


constraint, let alone be a feasible solution to (23.1). 


Lemma 24.5.1 [f B is a deficient basis, solution (24.46) satisfies the equality 
constraint of (23.1), corresponding to primal objective value f. 


Proof Note that all resulting tableaus represent the same problem (23.1). It is 
obvious that B is deficient basis if and only if k = m ork < m but bRR, = 0. 
Under the assumption, it is easily verified that solution (24.46) satisfies the equality 
constraint of (24.22), as well as that of (23.1), with the right-most entry of the 
bottom row giving the related objective value f. Oo 


Denote by Dyuy the dual face defined by (1.10) with N replaced by N’ UN. 
Lemma 24.5.2 If B is a deficient basis, Dyiun is a dual level face. 
Proof The validity of the statement is obvious through Corollary 24.1.2. oO 


Theorem 24.5.1 Assume that B is a deficient basis. If the following index set 
T ={ji € B|b; <0,i€ R} (24.47) 


is empty, then Dyyn is a dual optimal face with a pair of primal and dual optimal 
solutions. 


Proof Lemma 24.5.1 and emptiness of T together imply that x given by (24.46) is 
a primal feasible solution to (23.1). On the other hand, it is obvious that {yr,urR, = 
0, Z~’un = Cn’uN = 0} corresponds to 


{yr = Bp’ ce, yRuR. = 90, 28 = 0, 2wuN = Enun = 0} € Dyun, (24.48) 


which is a dual feasible solution. The preceding and x exhibit complementarity 
slackness and are therefore a pair of primal and dual optimal solutions. According 
to Lemma 24.5.2, Dyuy is a dual level face. Including the dual optimal solution, it 
is a dual optimal face. Oo 


In the case of T 4 4 when optimality cannot be declared, execute the following 
tactic, and go on the next iteration. 
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Tactic 24.5.1 Using T’ = {i € R| bj < 0}, update {B, N’, R, Ri, k} by 


= '=N'UT 

a a (24.49) 

R=R\T’, Ri = RUT kK=k-|TI. 

Consequently, we are faced with an expanded new dual face. Note that the zero- 

valued new Cy’ will never affect the determination of the next stepsize, since the 
new ba, is nonpositive, and hence Azjy = —bp, is nonnegative. 


24.6 Dual Face Algorithm: Tableau Form 


Based on the discussion made in previous sections, we formulate the dual face 
algorithm in tableau form in this section. 

In each iteration, the dual face tableau is updated with f increasing. A series of 
iterations are performed until k = m or b R,UR, = 0 when a dual level face is reached 
(unless upper unboundedness detected). At the end of such a series, referred to as 
dual face contraction, optimality is tested: if set T is empty, optimality is achieved. 
In the other case, the dual level face is expanded, and the next dual face contraction 
is carried out. 

Although the solution process can be initiated with any pseudo-basis, we prefer 
a deficient basis since this will make br, vanish throughout the whole solution 
process. Therefore, the simpler search direction formula (24.27) will be used. 

Let B be initial deficient basis. We take the following partition to get started: 


N= A\BN’=@, 


24.50 
R=R, R =O,k<m. ( ) 


The overall steps are summarized into the following model. 


Algorithm 24.6.1 [Dual face algorithm: tableau form] Initial: dual face tableau in 
form (24.22) with partition (24.50), br. = 0, and Cyun. This algorithm solves 
problem (23.1). 


1. Ifk =m or br, =0, then 


(1) Stop if T defined by (24.47) is empty 
(2) Execute Tactic 24.5.1 


. Compute Azniun by (24.27). 

. Stop if J defined by (24.29) is empty. 

. Determine B by (24.33). 

. If B > 0, update Cyn by (24.32). 

If l =N OJ! £G, where J' is defined by (24.38), execute Tactic 24.4.1, and 
go to step 1. 


NwnOKRWND 
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7. Determine column index q by (24.40) and row index p by (24.41). 
8. Ifap,q = 9, redetermine row index p by (24.43). 
9. Premultiply the p-th row by 1/@p,q, and then add —Gj,q times of the row to the 
i-th row, fori © Rj UR, iF p. 
10 If p € Re, execute Tactic 24.4.2; else execute Tactic 24.4.3. 
11. Go to step 1. 


Theorem 24.6.1 If each dual face contraction involves a nondegenerate dual 
feasible solution, Algorithm 23.5.1 terminates either at 


(i) Step 1(1), giving a pair of primal and dual optimal solutions, or at 
(2) Step 3, declaring primal infeasibility 


Proof It is seen that k increases by 1, at least, in each iteration. Therefore, within 
finitely many iterations, either k increases to m or b R\UR, Vanishes so that each dual 
face contraction ends with a dual level face (Lemma 24.5.2) unless detecting primal 
infeasibility at step 3, by Theorem 24.3.3. 

Assume that the algorithm does not terminate. Then, failing for optimality test, it 
performs infinitely many dual face contractions, each including a nonzero stepsize. 
Consequently, objective value f, corresponding to the end dual level face, increases 
strictly (Theorem 24.3.2), as contradicts that the number of the dual faces is finite. 
Therefore, the algorithm terminates, with outlet step 1(1) giving a dual optimal face 
together with a pair of dual and primal optimal solutions, by Theorem 24.5.1. oO 


24.7 Dual Face Algorithm 


In this section, we convert Algorithm 24.6.1 into a revised form using LU 
factorization. 

Introduce symbol B = BUN’. Let current Cyyn, b, and LU factorization below 
be available: 


Brur, = LU. (24.51) 


The following quantity expressions required can be derived based on a compari- 
son between (24.8) and (24.13), with B and R replaced, respectively, by BU N’ and 
RU Rj, e.g., giving 


Nrur, = ee NRuRi: Nr. = Nr. - Br. Bair, NRuRi (24.52) 


which are useful for obtaining the search direction. 


(A) Search direction Azyuy = {—br,, —N} bp, — NE bp, } (24.25). 


‘ A = coat 
@ t] = 52 Np, = OF, 62 Babe, Neur: 
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Solve following triangular systems for v = Bae R ol, bp yt; 


Jt 
uTu=(0f,6h) , Liv=u, (24.53) 


and then compute 
t = Naya, (24.54) 


wy, AT? ~~ — wei 
(i) » = Np bp, = (Nr, — Br. Brig, Nrur,) br, (ignore if bp, = 0). 


Solve the following systems for w = Brie Be br. first: 


Ulu=Bhor, L'w=u, (24.55) 
and then compute 
tp = Ni br, — Nur, wv. (24.56) 
Finally, it follows that 
Azn = {—br,, —t — ty}. (24.57) 


(B) Pivot subcolumns {@RuUR,,q, 4R,,q> 4R.,q}- _ 
Solve the following triangular systems for w = G@RuUR,.q = Ban GRUR},.q! 


Lu = @RuR\,q> Uw=u. (24.58) 
Then it follows that 
GRiq = WR} Req = AReq — BR. (24.59) 


(C) Update of right-hand side b (ignore if p € Re and br. = 0). 
Determine aw = b p/@p,q, and update 


= A 


b=b-—aa, by=a. (24.60) 
As for updating LU factors of Brur, nothing has to do with Tactic 24.4.1 
or 24.5.1, but the following two cases arise: 


Case 1 Rank increasing with Tactic 24.4.2: rank k of Rag R, increases by | (see 
Sect. 19.4.2). 
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Case 2 Rank remaining with Tactic 24.4.3: k remains unchanged. The related 
updating is the same as in conventional simplex context, in essence. 


Now Algorithm 24.6.1 can be converted into the following model. 


Algorithm 24.7.1 [Dual face algorithm with LU factorization] Initial: deficient- 
basis B, partition (24.50), Brur, = = LU, brur: and Cyuy. This algorithm solves 
problem (23.1). 


. Go to step 4ifk =m. 

. Goto step 6 if Azy = —br, £0. 

. Stop if T defined by (24.47) is empty (optimality achieved). 

. Execute Tactic 24.5.1, and go to step 2. 

. Solve systems (24.53) for v, and compute Azy = —Nior, v. 

. Stop if J defined by (24.29) is empty (infeasible problem). 

. Determine 6 by (24.33). 

. If 6 > 0, update cyruy by (24.32). 

_ If J =N'NJ' ZG, where J’ is defined by (24.38), execute Tactic 24.4.1, and 
go to step 1. 

. Determine column index qg by (24.40). 

. Solve systems (24.58) for w = GRuUR,,q- 

. Compute dr,,, by the second expression of (24.59), and determine p 
by (24.41). 

. Ifdp,g #0, then 


(1) Update L, U by Case 1, and execute Tactic 24.4.2 
else 
(2) Redetermine p (with j,) by (24.43) 
(3) Update L, U by Case 2 
(4) Update brur, by (24.60), where a = bp /ap,q, and execute Tactic 24.4.3 
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14. Go to step 1. 


24.8 Notes on Dual Face Method with LU Factorization 


If the proposed algorithm gets started from a pseudo-basis rather than deficient 
basis, one should use the search direction (24.25) instead of (24.27), since the 
nonzero b Rr, then contributes extra growth 6 \|b Re ||? to the dual objective increment, 
by Theorem 24.3.2. Accordingly, however, two more triangular systems have to be 
solved. It might be a good idea to get started with a pseudo-basis of small k, proceed 
with N’, R; = G, and search direction 


Azy = —NE Gr, /llbR ll); (24.61) 


until b R, Vanishes, and go over to Algorithm 23.6.1 with direction (24.27) instead. 
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Alternatively, instead of the whole b R,, it is possible to use a part of it to form 
search direction, e.g., such that 


lbil = p, ie Ry, (24.62) 
where p is a threshold, decreasing in the solution process. 

Surprisingly enough, it turns out that Algorithm 24.7.1 and Algorithm 23.6.1 are 
just a pigeon pair. It is interesting to compare the two algorithms. It seems to be 
natural to use the latter to achieve dual feasibility and then use the former to achieve 
optimality, or vice versa. It is noted that both algorithms can be implemented, the 
same as in the conventional simplex context. 

An advantage of the new dual face algorithm is that it solves small systems, 
compared with those solved by the simplex algorithms. In a single iteration, 
therefore, the overall computational cost by the former is less than the latter. 

Another advantage of the new algorithm is its good stability, compared with 
simplex algorithms. The former tends to select a large pivot in magnitude, while the 
latter’s pivot can be arbitrarily close to zero. As we all know, the latter sometimes 
fails to maintain a usable basis and has to restart from scratch. 

Finally, the most attractive feature of the proposed algorithm might be the dual 
objective increment 6 \|b R ||? achieved per iteration. It is expected that this would 
significantly reduce the number of iterations required. At this stage, however, there 
are no numerical results available. We do not even know which search direction 
is better in practice. The implementation of the new algorithm and computational 
experiments are expected. 
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Chapter 25 ® 
Simplex Interior-Point Method od 


To cope with degeneracy, the face (or dual face) method introduced previously does 
not seek the optimal solution, vertex by vertex, but instead, face by face. However, 
it still faces degeneracy. It seems that the degeneracy is inevitable if the iterates are 
confined on the boundary of the feasible region. This leads us again to the interior- 
point method, approaching the optimal solution by moving across the interior of the 
feasible region. As it is known, nevertheless, the latter has been seriously restricted 
in applications because it cannot be warmly started. To obtain the advantages of 
both, therefore, it is very attractive to combine the two types of methods into one. 
There is no insurmountable gap between them even if they look different. In 
this chapter, we show that the combination is achievable. In principle, any simplex 
algorithm can be transformed into an interior-point algorithm by appropriately 
modifying the pivot rule and the termination criterion and shortening the stepsize. 
The so-called simplex interior-point algorithm, introduced in this chapter, is 
transformed from the conventional generalized simplex algorithm. Starting from 
an interior point, it generates a series of interior points to achieve the optimality. 
In the next section, we present the column rule and search direction first. In 
the subsequent section, we address the row rule and stepsize. In the remaining 
sections, we highlight the optimality criterion, formulate the algorithm, and report 
encouraging computational results obtained from our extensive experiments. 
Consider the bounded-variable problem below: 


(25.1) 


where A € R”*"(m <n); rank A = m. The values of components of / and u are 
assumed to be finite. For problems with infinite bounds, one may take numbers of 
large enough module instead. 
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Let B be the current (normal) basis and N the associated nonbasis, 1.e., 
B={ji,.--, jm}; N=A\B. 


Note that j; is the index of the ith column of B or the basic index corresponding to 
the ith row of the associated simplex tableau. 


25.1 Column Pivot Rule and Search Direction 


Let x be the current interior solution. The reduced cost is 


ép = 0, én = cy —N'5, B'y =cg. (25.2) 


The normal generalized simplex algorithm selects an entering index based on ¢ 
only. In contrast, we will not only consider c but also the possible ranges that the 
nonbasic components of the solution are allowed to change. 

To begin with, introduce 


Xj — Uj, ifc; <0, 
sj;=4x;-l), if@;>0, jen, (25.3) 
0, if ¢; = 0, 


which is the distance from x; to one of the associated bounds that is violated if x; 
changes to reduce the objective value. If c; 4 0, then 6;c; > O since x is an interior 
solution. 

The following rule is the key to the proposed algorithm. 


Rule 25.1.1 (Column Pivot Rule) Select a nonbasic index q by 


qe nee Cj 5). (25.4) 


Then, the following vector Ax = {Ag, Aj} can be determined: 


Axp = Sign(Cg)aq, Bag = dq. (25.5) 
0, JEN; j#q 
Axj = (25.6) 
—sign(Cq), jJ=4. 


Using the preceding notation, the following lemma reveals the eligibility of Ax 
to be a search direction. 
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Lemma 25.1.1 Assume that q is determined by Rule 25.1.1. If Cq5q > 0, the vector 
Ax is a descent direction in the null of A with respect to the objective. 


Proof Note that cg 4 0. 
From (25.5)—(25.6), it follows that 


AAx = sign(é,) BB”! ag — sign(Cg)dg = 0. (25.7) 


Thus, Ax is in the null of A. 
Furthermore, it holds by (25.2) that 


aa T p=1 
Cq =Cq — CRB ag, 


which together with (25.5), (25.6), and (25.4) gives 


c' Ax = —sign(é,)cq + sign(ég)cpB ‘dq 
= —sign(,) (cq — cg B~'ay) 
= —sign(Cg)Cq 
= —|Cq| 
<0. (25.8) 
Therefore, Ax is a descent direction with respect to the objective function. oO 


25.2 Row Pivot Rule and Stepsize 


Using Ax as search direction, we form the normal line search below: 


X=xt+aAx, (25.9) 
or equivalently, 
Xp =xXptadAxp, (25.10) 
Xq = Xq — sign(C,)o, (25.11) 
Kj =xj, JEN; j#a, (25.12) 


where @ is the stepsize to be determined. 
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Introduce 


(uj, —xj,)/Ax;, if Ax;, > 0, 
oj = i=l,...,m, (25.13) 
(Lj; = Xj) / AX j; if AX j < 0, 
which is the maximum range that x, is allowed to change while keeping x;, within 


its bounds. Considering all rows, we are led to the following rule for selecting row 
index. 


Rule 25.2.1 (Row Pivot Rule) Determine row index p such that 
Oy = min {oj |i = 1,..., m}. (25.14) 
Therefore, the largest possible stepsize subject to] < x < u is 
a = min {dg, op} > 0, (25.15) 
where 4; is defined by (25.3). 


Then, there are two cases arising: 


(i) 5g < op. There is no need for any basis change. 
(ii) 6g > Op. The basis change should be performed with leaving index j, and 
entering index q. 


For the new solution < to be interior, however, we do not use @ itself but a 
shortened one instead, 1.e., 


a=ra, OK<aA<]l, (25.16) 


where A is a predetermined parameter close to 1. 
Then, the following result can be claimed. 


Lemma 25.2.1 Let x be an interior solution. If Cq5q > 0, the x defined by (25.10)— 
(25.12) with the a given by (25.16) together with (25.13)-(25.15) is an interior 
solution with the objective value reduced by X4Cq > 0. 


Proof Since x is an interior solution, it can be verified that t defined by (25.10)- 
(25.12) satisfies equality constraints AX = b for any real number q, in particular for 
the a determined by (25.16). Furthermore, from (25.10)-(25.16), it follows that x 
satisfies 1 < x < u and is hence an interior solution. 

Moreover, from (25.9) with (25.8) and (25.16), it follows that 


cl —clx = acl Ax = —Aalé,| <0, (25.17) 


and the claim is therefore valid. oO 
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25.3 Optimality Condition and the Algorithm 


The search direction Ax is useless if C,5, vanishes. In this case, the optimality 
condition is fulfilled, as the following says. 


Theorem 25.3.1 The current x is an optimal solution if and only if it holds that 
Cq5q = 0. (25.18) 


Proof Necessity. Assume that x is an optimal solution. If (25.18) does not hold, i-e., 
Cq5q > 0, then new solution x can be determined with strictly lower objective value 
(Lemma 25.2.1), leading to a contradiction. 

Sufficiency. By (25.4), it is clear that condition (25.18) is equivalent to 


éjsj=0, VEN. (25.19) 


For each j € N, the quantity c;6; is the upper bound of the amount by which the 
objective value can reduce as the value of x; changes toward the associated bound. 
Therefore, (25.19) implies that the objective value cannot decrease any further, and 
the claim is therefore valid. Oo 


Note that the preceding theorem can also be derived from the optimal condition 
for the bounded-variable problem (Theorem 7.3.4 of Part I). 

Instead of (25.18), however, we will use the following approximate optimality 
condition. 


Definition 25.3.1 The current solution x is an €-optimal solution if 
lcjl<e1 or |d;| <2, VIEN; (25.20) 


where €; > 0 and €2 > 0 are predetermined tolerances. 


Correspondingly, 6; given by (25.3) will be replaced by 


Xj — Uj, ife; < —-€1, 
0, otherwise, 


The overall steps can be summarized into the following algorithm. 


Algorithm 25.3.1 (Simplex Interior-Point Algorithm) Given 0 <  < 1, 
€1, €2 > Oand M > 1. Initial: basis B, associated with interior solution x. This 
algorithm solves the bounded-variable problem. 
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. Solve B'¥ = cg for ¥. 

Compute Cy = cn — N'y. 

Compute 5;, j € N by (25.21). 

. Stop if (25.20) is satisfied. 

. Select column index q by q € argmaxjen C;45;. 
. Solve Bag = dg for aq. 

. Compute Ax by (25.5) and (25.6). 

Determine row index p by (25.14) with (25.13). 
. Stop ifo, > M. 

. Seta = ha, where a = min {bg, op} 

. Update x by (25.10)-(25.12). 

. Goto step I if 5g < op. 

. Update basis B by replacing its jp-indexed column with q-indexed column. 
. Go to step 1. 


SA NON 
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Concerning meanings of the exits of the preceding, the following can be claimed. 


Theorem 25.3.2 Assume the finiteness of Algorithm 25.3.1. It terminates either 
at 


(i) Step 4, producing an €-optimal basic solution, or 
(ii) Step 9, declaring the lower unboundedness of the problem 


Proof The meaning of the exit step 4 is based on Theorem 25.3.1 and Defini- 
tion 25.3.1 and that of step 9 is obvious. Oo 


In some applications, where the required is a convex rather than an interior 
solution, the following simple purification tactic can be incorporated into Algo- 
rithm 25.3.1: 

Assume that Algorithm 25.3.1 terminates at step 4 with an €-optimal solution. We 
move nonbasic components of the solution onto their nearest bounds, respectively, 
maintaining basic components unchanged. If the resulting solution, say x°, satisfies 
Ax° = b within some tolerance, it can be viewed as optimal convex solution. In the 
other case, a standard two-phase generalized simplex algorithm applies to attaining 
optimal convex solution, hopefully within few iterations. 


Remarks 

Algorithm 25.3.1 can get itself started from any feasible point, not just interior point. 
Moreover, if the first stepsize taken is positive, it generates a series of interior points 
subsequently. 

The method of this type actually minimizes the objective function over a feasible 
region shrunken from the original feasible region but uses a special termination 
criterion. In each iteration, it still makes a basis change but uses a special pivot rule 
and takes a positive stepsize. 
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Table 25.1 Ratio 


MINOS/FPSP MINOS/FPS FPSP/FPS 
Problem Itns Time Itns Time Itns | Time 
Kennington(16) 6.6 32 9.0 3.5 1.4 1.1 
BPMPD(15) 2.8 3.4 9.5 9.0 3.3 2.6 
Average(31) 3.3 3.4 9.4 6.6 2.8 2.0 


25.4 Computational Results 


Extensive computational experiments have been performed to gain an insight into 
the behavior of Algorithm 25.3.1. A summary of the associated results is offered in 
this section. 

Implemented and compared are the following three codes: 


. MINOS: MINOS 5.51 with full pricing 
. FPS: two-phase code based on Algorithm 25.3.1 
3. FPSP: two-phase code based on Algorithm 25.3.1 with the purification 


Noe 


The first set of test problems included all 16 problems from Kennington and 
the second included all 17 problems from BPMPD that were more than 5O0KB in 
compressed form (Appendix C: Tables C.2—C.3). 

In Table 25.1, a comparison between the three codes is made (see Appendix F 
for details). 

From the preceding table, it is seen that FPSP and FPS outperformed MINOS 
remarkably, with average iteration ratios 6.6 and 9.0 and time ratios 3.2 and 3.5 
for the 16 Kennington problems. They outperformed MINOS, by average iteration 
ratios 2.8 and 9.5 and time ratios 3.4 and 9.0 for BPMPD problems. For the entire 
set of the 31 test problems, FPSP and FPS defeated MINOS by average iterations 
ratio 3.3 and 9.4 and time ratios 3.4 and 6.6. 

In summary, the simplex interior-point algorithm much overperforms the simplex 
algorithm. 


Chapter 26 ®) 
Facial Interior-Point Method Ghost for 


The simplex interior-point method, introduced in the previous chapter, is a combi- 
nation of the simplex method and the normal interior-point method. In contrast, the 
so-called facial interior-point method proposed in this chapter is a combination of 
the face method and the normal interior-point method. 

Any normal interior-point algorithm can be transformed into such a type, in 
principle. Starting from an interior point (in the n-dimensional face), the new 
algorithm moves along the same search direction to the nearest boundary and takes 
a scaled shortened stepsize, attaining an interior point in a lower-dimensional face. 
Such an iteration is termed facial iteration. Then, it performs another facial iteration 
by moving along the direction projected onto the face to the nearest boundary and 
takes a scaledly shorten stepsize, and so on, until approaching optimality (according 
to appropriate termination criteria) or performing another series of facial iterations. 

Along this line, in this chapter, we spend three sections to introduce the facial 
affine face interior-point algorithm, facial D-reduced interior-point algorithm, and 
facial affine interior-point algorithm, respectively. 


26.1 Facial Affine Face Interior-Point Algorithm 


The search direction utilized in a normal interior-point algorithm is characterized by 
a combination of the steepest descent direction and a so-called centering direction 
towards the interior of the feasible region. The search direction in the affine face 
Algorithm 21.7.1 is just such a direction. In this section, therefore, we transform the 
algorithm into a normal interior-point algorithm first and then transform the latter 
further into the desired algorithm by merging the face algorithm. 
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Return to the reduced problem (21.1). Initially, the search direction is given 
by (21.25) withk =n+1lorB=A,ie., 


A=—eny1 + X?ATy, — (AX?A™)¥ = —em41, (26.1) 
where 
X = diag(X1,..., Xp, 1). (26.2) 


What needs to do is to start from an interior point and take 1 times of the original 
stepsize, where 0 < 4 < 1 is close to | (e.g., 95%~99%). 

Using the preceding notation, the overall steps can be put into the following 
algorithm. 


Algorithm 26.1.1 (Affine Face Interior-Point Algorithm) Given e < 1 andi € 
(0, 1) close to 1. Initial: an interior point x > 0. This algorithm approximately 
solves the reduced problem (21.1). 


. Compute AX? AT = LL". 

. Solve LTy = —(1/v)em+1 for y, where v is the (m + 1)th diagonal of L. 
. Compute A = —en41; + X7A'Y. 

. Stop if || A|| < €. (optimality achieved) 

. Stop if J = {j ¢ A| Aj < 0} = M. (lower unbounded) 

. Determine a = Aminjey —X;/Aj. 

. Update: x =x +aA. 

. Go to step 1. 


OANANDNPWN Ke 


The author was astonished that the preceding algorithm is the same as the Dikin’s 
algorithm (Algorithm 26.3.1) in essence. The only difference is that the former 
solves the reduced problem while the latter solves the standard problem. Therefore, 
if Dikin had handled the reduced problem rather than the standard problem, he 
would have gotten Algorithm 26.1.1. 

It should be indicated that Algorithm 26.1.1 is preferable from a computational 
point of view, since it solves one triangular system less in each iteration. 

Further, Algorithm 26.1.1 can be transformed into a facial interior-point algo- 
rithm by merging the face algorithm. 


Algorithm 26.1.2 (Facial Affine Interior-Point Algorithm) Given A ¢€ (0, 1). 
Initial: interior point x > 0. This algorithm solves the reduced problem (21.1). 


1. Setk=n+1,B =A, N = @ and compute Cholesky factorization BX, BT = 
LL", where Xz is defined by (21.27). 

. Solves LTY = —(1/v)em+1, where v is the (m + 1)th diagonal of L. 

. Compute Ag = —ex + XL By. 

. Go to step Ll if Ag = 0. 

. Stopif J = {j ¢ B| A; <0, 7 An +1} = O (ower unbounded problem). 


nb wWh 


26.1 


6. 


7. 
8. 
9. 
10. 
11. 
12. 
13. 
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Determine the stepsize a and index p such that a = —xX),/Ap = 
MiNjey —x;/Aj. 

updatexg = Xp + AaAg. 

Update (B, N) by bringing p from B to N. 

Update L by Algorithm 21.3.1. 

Setk =k —1;ifk > m+ 1, go to step 2. 


Compute Zy = —NTy. 
Stop if zy > 0 (optimality achieved). 
Go to step 1. 


Example 26.1.1 Solve the following problem by Algorithm 26.1.2: 


min f 
s.t. —4x, +3x2 +23 —2x4 +2x5 = 5, 
3x, —Xx2 +2x3 —3x4 —4x5 = —8, 
Xp +x2 +2x3 +24 +3x5 = 12, 
—2x] —3x3 42x, +x5-f= 0, 


np 20, f= tyx.2,6, 


Set A = 99/100. Initial interior point :x = (1, 2, 1, 1, 2, -1)". 


Answer Iteration 1: 


1. 


Nn 


k = 6,B = {1,2,3,4,5,6}, N = @. Xz = diag(1, 2, 1,1, 2,—1)"; face 
—-4 3 1-2 2 

3 -1 2 -—3-4 

1 1 2 1 3 

—2 —3 2 1 -!1 


matrix B and the Cholesky factor of BX 2 B’ are B = 


1399/134 
~2362/411 7176/919 

1985/471  —1327/373 916/207 
1181/822 —302/85 —800/331 815/672 


L= 


. ¥ = (—100/5011, —2485/5193, —215/579, —1219/1793)7, 
Bp = (—583/1587,—2127/2168,509/1591,—356/1393,278/865, 


—1385/3339)' 4 0. 


_J=(1,2,4 49. 


a = (99/100) min{—1/(—583/1587, —2/(—2127/2168), —1/(—356/1393)} 
= (99/100)(1745/856) = 2333/1156, p = 2. 
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¥p = (1,2, 1,1,2,-)" 
+ (2333/1156)(—583/1587, —2127/2168, 509/1591, —356/1393, 278/865, 
— 1385/3339)! 
= (323/1249, 750/37499, 1593/968, 261/539, 2005/757, —2019/1099)!. 
8. B= {1,3,4,5, 6}, N = {2}. 
882/145 
290/49 307/43 


1450/441 —959/279  3244/737 
145/98 —7775/2199 —1184/473 2500/2447 


9 L= 


10. k=5. 
Iteration 2: 
2. y = (—361/1907, —1405/1907, —1039/1907, —845/882)". 


3. Ag = (—156/1907, 232/1907, 244/1907, —184/1907, —277/6603)' + 0. 
5. J={l, 5X. 
6. 


a = (99/100) min{—(323/1249) /(—156/1907), —(2005/757) /(— 184/1907)} 
= (99/100)(7761/2455) = 917/293, p= 1. 


XB = (323/1249, 1593/968, 261/539, 2005/757, —2019/1099)? 
+ (917/293) (—156/1907, 232/1907, 244/1907, —184/1907, —277/6603)" 


= (109/42149, 1688/833, 583/659, 1117/476, —5360/2723)!. 


8. B= {3,4,5, 6}, N = {1, 2}. 

3524/769 
—1435/274 1211/172 

1435/274 —3433/1235 1187/491 

769/3524 —2201/577 —499/322 1 
10..k=4=m+1. 
11. Zy = (156/1907, 717/1907)! > 0. 
12. The approximate basic optimal solution and optimal value are 


ae 


¥ = (0.0026, 0.0200, 2.0264, 0.8847, 2.3466)', f = —1.9684. 
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The outcome is close to the exact basic optimal solution and optimal value, i.e., 
x* = (0, 0, 2.0256, 0.8718, 2.3590)", f* = —1.9744. 


The maximum error in components of xg is 0.02, and that in f is 0.006. 


26.2 Facial D-Reduced Interior-Point Algorithm 


The search direction, used in the D-reduced gradient algorithm (Sect. 20.5), is uphill 
for the dual objective, and points to the interior of the feasible region. Therefore, the 
direction is suitable for establishing a normal interior-point algorithm. 

The overall steps are put in the following algorithm using the deficient basis. 


Algorithm 26.2.1 (D-reduced Interior-Point Algorithm) Given w > 0, A &€ 
(0,1), € > O. Initial: interior point z > 0. This algorithm solves the D-reduced 
problem 17.1. 


1. Compute w by (20.15). 

2. Stop if J = {j € A|w; > 0} = Y (infeasible problem). 
3. Compute B = Z,/@g = MiNjey Z;/@j. 

4. Stop if B < € (approximate optimality achieved). 

5. If B ~ 0, update z = z — ABo. 

6. Go to step 1. 


As in the previous section, we transform the preceding algorithm into the 
following facial interior-point algorithm by merging the face algorithm. 


Algorithm 26.2.2 (Facial D-reduced Interior-Point Algorithm) Given 4 > 
0, A € (0, 1). initial :zy > 0. This algorithm solves D-reduced problem 17.1. 


1. Set k = Oand give B, R, N, R’ by (20.6). 
. Compute oy = Nlen — UNL Ney), and set Oy = wy. 
. Stop if J = {j € N | @; > 0} = GY (infeasible problem). 
. Determine £ and q such that B = Zg/@g = ey z;/@;, and compute AB. 


2 

3 

4 

5. If B £0, update zy = Zy — ABon. 

6. Compute a,(R) = Br dq(R), g(R’) = ag(R’) — Briag(R). 
7. Go to step 12 if R’ = M or ag(R’) = 0. 

8. Determine row index p € arg maxjeR’ |G; ql. 

9. Update Bz’ by (19.10). 
10. Setk =k + 1, bring q from N to B and p from R’ to R, 
11. Compute ®y = wy — NI(BR’ ws), and go to step 3. 


12. Determine s € arg max{dj,.¢ |t=1,...,k}. 
13. Go to step 1 if aj, g > 0. a 
14. Compute x, = 1/@g, XB = —Xqdq(R), f = cqXq + Ch aD: 


15. Stop (optimality achieved ). 
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Example 26.2.1 Solve the following problem by Algorithm 26.2.2: 


min $f = 2x; +x2+3x3+ x4 + 2x5 + 4X6, 
s.t. x; +3x2 —2x3 +3x4 —4x5 +2x6 = 0, 


—2x2 —x4 —2x5 +x6 = 0, 
—2x, +x2 +2x3 —4x4 +3x5 —3x6 = 0, 
+2x2 +3x3 —2x5 —-x6 = 1, 


xj>0, j=Hl,...,6. 


Answer For the convenience of comparison, the related tableau will be given for 
each iteration. 


Initial tableau: 


pal x2 x3 X4 X5 x6 | RHS 
1 3 -2 3 -4 2 
—2 -1l -2 1 
—2 1 2 —4 3. =3 
2 3 —2 -l 1 


Iteration 1: 


1.k=0; B,R=6, N=({l,...,6}, R’ = {1, 2,3}. Zw = (2, 1,3,1,2,4)". 
2. 


on = (0, 2,3, 0, —2, -1)? — NE.Ne (1/2, 1, 1/3, 1, 1/2, 1/4)? 
= (—19/2, —191/12, 101/6, —325/12, 187/12, —41/3)', @y = on. 
3. J = (3, 5}. 
4, 
B = (95/100) min{3/(101/6), 2/(187/12)} = (95/100) (24/187) 
= 114/935, q =5. 
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—19/2 


x3 x4 
3 =2 3 
—2 -1 
1 2 —4 


191/12 101/6 325/12 


1 3 1 


Zw = (2, 1,3, 1,2, 4)? 


x5 RHS 


187/12 
2 
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— (114/935)(—19/2, —191/12, 101/6, —325/12, 187/12, —41/3)!. 


Sow eMANID 


_ 


-1/4 
hyo 
—5/4 

—269/48 

1696/537 


Iteration 2: 


3 JS 1By, 
4. 


. Gg(R’) = (—4, —2, 3)7. 

. Gg(R’) £0. - 
p=l1,1=-1/4, B, = (—1/4). 
_k=1, B= {5}, R= {I}. N = (1, 2,3, 4, 6}, R’ = {2, 3}. 


&v = (—19/2, —191/12, 101/6, —325/12, —41/3)" 


— (1,3, —2, 3, 2)'(—1/4) (187/12) 


= (—269/48, —203/48, 217/24, -739/48, —47/8)". 


x2 
—3/4 
S775 
13/4 
~203/48 
644/219 


x3 x4 
1/2 —3/4 
1 S578 
1/2 1a 


217/24  —739/48 
886/935 1609/374 


x5 X6 RHS 
1 —1/2 
—3/2 
—47/8 1 
1/10 = 5298/935 | —114/935 


B = (95/100) min{ (886/935) /(217/24)} = (95/100) (875/8349) 
= 205/2059, q =3. 
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Zy = (1696/537, 644/219, 886/935, 1609/374, 5298/935)* 
— (205/2059)(—269/48, —203/48, 217/24, —739/48, —47/8)" 
= (1349/363, 9359/2784, 47/992, 4102/703, 2513/402)". 


6. Gg(R) = (—1/4)(—2) = 1/2, Gqg(R’) = (0, 2)? — (—2, 3)1(—2) = (—4, 8)". 
. Gg(R’) #0. 


~ 


p=3, t= (2-3(1/2))"! =2, v = —2(1/2) =-1, d? = —2(3)(-1/4) = 3/2. 


- n eG i he 
U=(-1/4)—(-D3(-1/4)=—-1, Be aS a 


9.k=2, B= {5,3}, R= {1,3}. N = {1,2,4, 6}, R’ = {2}. 


on = (—19/2, —191/12, —325/12, —41/3)" 


T T 
13 3 2 = =7 
= ‘s an 3) iG, : (187/12, 101/6) 


= (17, —63, 65/4, 85/4)". 


x1 x2 X3 X4 x5 X6 RHS 
1 —4 1 1 1 
2 —10 1 3 
5/2 13/2 1 7/2 3 
17 63 65/4 85/4 1 


1349/363 9359/2784 47/992  4102/703 1/10  2513/402 | —905/4086 


Iteration 3: 
3. J = {1,4, 6}. 
4. 
AB = (95/100) min{(1696/537)/17, (4102/703) /(65/4), (2513 /402)/(85/4)} 
= (95/100) (473/2546) = 473/2680, g = 1. 


26.2 Facial D-Reduced Interior-Point Algorithm 


5: 
Zn = (1349/363, 9359/2784, 4102/703, 2513/402)" 
— (473/2680)(17, —63, 65/4, 85/4)" 
= (577/806, 9398/649, 1528/515, 1608/643)". 

6. 

iW(®)=( 529) (-2) = (sa): 

Gg(R’) = 0 — (—2, 0)(1, —2)? = 2. 
7. Gq(R’) # 0. 
8. 


p=2.t = (0- (-2, 0)(1, —5/2)")7! = 1/2. 
vy = —(1/2)(1, —5/2)? = (-1/2, 5/4)". 
d™ = —(1/2)((—2, 0) Bg!) = —(1/2)(2, 2) = (-1, -D. 


af ES fy, T “i 0 0 
cae ) (—1/2, 5/4) oa an) 


0 0-1/2 
B,! = =f =1/2° 5/4 1). 
SL: St 12 
9. k=2, B={5,3, 1}, R={1,3,2}.N = (2,4, 6}, R=. 
10. 


@v = (—191/12, —325/12, —41/3)" 


ON PT “eo 
ge ere 1-1/2 5/4) (187/12, 101/6, —19/2)" 
-1 ] = 


-2 |. Sal 


= (22, 31/4, -17/4). 
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577/806 


Iteration 4: 


3. J = 2,4). 
4. 


x2 


9398/649 47/992 


x3 


X4 
1/2 
1/2 
—9/4 
31/4 
1528/515 
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x5 X6 RHS 
1 1/2 
3/2 
3/4 
17/4 1 
1/10 = =1608/643 | —1143/2872 


AB = (95/100) min{(8645 /597)/22, (1528/515)/(31/4)} 
= (95/100)(1557/4067) = 559/1537, g = 4. 


Zn = (9398/649, 1528/515, 1608/643)' — (559/1537)(22, 31/4, —17/4)! 


= (6460/997, 301/2029, 3395/839)!, 


577/806 


6. Gq(R) = 


= 
7. Gq(R') = 0. 


x2 


-1 


x3 


-5 
6-4 
22 
6460/997 47/992 
0 0-1/2 
Lal oi 


1/2 


-1 


x5 x6 RHS 
1 =12 
3/2 
3/4 
—17/4 1 
1/10 3395/839 | —636/835 
1/2 
-9/4 
1/2 


11. max{1/2, -9/4, 1/2}=1/2>0, s=1, p=. 
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Iteration 5: 


1.k=0; B,R=9%, N=(l,...,6}, R’ = (1,2, 3}. 


2. 
on = (0, 2, 3,0, -2,-1)7 
_ Np Np (806/577, 997 /6460, 992/47, 2029/301, 10, 839/3395)! 
= (9606/67, 1965/23, —22001/110, 22051/69, —23926/57, 6767/25)", 
ON = ON. 
x1 x2 X3 X4 x5 x6 RHS 
1 3 —2 3 —4 2 
—2 -—1 —2 

—2 1 2 —4 3 3 
9606/67 1965/23 —22001/110 22051/69 —23926/57 6767/25 1 
577/806 6460/997 47/992 301/2029 1/10 3395/839 | —636/835 


3, J ={1,2, 4, 6}. 
4, 
AB = (95/100) min{(577/806) /(9606/67), (6460/997)/(1965/23), 
(301/2029)/(22051/69), (3395/839) /(6767/25)} 
= (95/100)(37/79707) = 8/18141, g = 4. 


Zn = (577/806, 6460/997, 47/992, 301/2029, 1/10, 3395/839)" 
— (8/18141) (9606/67, 1965/23, —22001/110, 22051/69, 
— 23926/57, 6767/25)" 
= (295/452, 1604/249, 189/1394, 104/14021, 1074/3767, 1347/343)!. 
. Gg(R’) = 3, -1, -4)7. 
Gq (R’) #0. 


p=3.t=—1/4. By! = (-1/4). 
_k=1, B= {4}, R= {3}. N =({1,2,3,5,6}, R’ = {I, 2}. 


oo Na 
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10. 
Ov = (9606/67, 1965/23, —22001/110, —23926/57, 6767/25)" 
— (—2, 1, 2, 3, —3)'(—1/4)(22051/69) 
= (—25331/1543, 15045/91, —10457/260, —7743/43, 6478/209)!. 

x1 x2 X3 X4 X5 X6 RHS 
—1/2 15/4 —1/2 7/4 —1/4 

1/2 —9/4 —1/2 11/4 7/4 

1/2 -1/4 —1/2 1 —3/4 3/4 

—25331/1543 15045/91 —10457/260 —77143/43 6478/209 1 


295/452 1604/249 189/1394 104/14021 1074/3767 1347/343 | —1022/1341 


Iteration 6: 


3, FS 0.6) 


B = (95/100) min{(1604/249) /(15045/91), (1347/343) /(6478/209)} 
= (95/100)(239/6134) = 311/8402, g =2. 


5. 
Zu = (295/452, 1604/249, 189/1394, 1074/3767, 1347/343)! 
— (311/8402)(—25331/1543, 15045/91, —10457/260, 
— 7743/43, 6478/209)! 
= (1893/1502, 439/1363, 2019/1243, 1821/262, 1957/704)'. 
6. Gg(R) = (-1/4)() = 1/4, aqg(R') = (3, -2)' — B,-1)1(-1/4) = 


(15/4, —9/4)?. 
7. Gg(R’) #0. 
8. 


p=1,1=@-3(-1/4)7! =4/15, v = —(4/15)(-1/4) = 1/15, 
d' = —(4/15)(3)(—1/4) = 1/5. 


Loin ae = (1/51/15 
U = (—1/4) — (1/15)3(-1/4) = -1/5, BB a ayeue) 
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9. k=2, B= {4,2}, R= {3,1}. N = {1,3,5, 6}, R’ = {2}. 
10. 


@v = (9606/67, —22001/110, —23926/57, 6767/25)" 


T T 
20). 2B S3\ fs 7 
= 22051/69, 1965/23 

( (eae eee ) eee, ( / on 


= (1570/279, —1763/97, —9777/95, 14664/349)". 


x] x2 X3 x4 X5 X6 RHS 
—2/15 1 —2/15 -7/15 -1/15 
1/5 —4/5 —19/5 8/5 
T/15 -8/15 1 —13/15 11/15 
1570/279 —1763/97 —9777/95  14664/349 1 


1893/1502 439/1363 2019/1243 104/14021 1821/262 1957/704 | —553/692 


Iteration 7: 
3. J = {1, 6}. 
4. 
AB = (95/100) min{ (1893/1502) /(1570/279), (1957/704) /(14664/349)} 
= (95/100) (426/6439) = 436/6937, gq = 6. 


5. 
Zy = (1893/1502, 2019/1243, 1821/262, 1957/704)? 
— (436/6937)(1570/279, —1763/97, —9777/95, 14664/349)" 
= (437/482, 3118/1127, 5287/394, 113/813)'. 
6. 


— (-1/5 1/15\ (-3\ _ [ 11/15 
ay) = (57/8 ais) ( = (ans) 
Gg(R’) = 1 — (—1, —2)(11/15, —1/15)" = 8/5. 


7. Gig(R’) £0. 
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p=2, t=(1—(-1, -2)(11/15, —1/15)")7! = 5/8. 


v = —(5/8)(11/15, —1/15)? = (—11/24, 1/24)". 
d" = —(5/8)((—1, —2) Bz!) = —G/8)(—1/5, —3/5) = (1/8, 3/8). 


U= i 2 — (—11/24, 1/24)' (-1/5, —3/5) = ee a) : 


1/15 4/15 5/24 7/24 


~7/24 —5/24 -11/24 
By'=| 5/24 7/24 1/24 
1/8 3/8 5/8 


9. k=2, B={4,2,6}, R = (3, 1,2}.N = {1,3, 5}, RB’ =9. 
10. 


@Nv = (9606/67, —22001/110, —23926/57)" 
20. 0 BN oe Bata 
= “qo 24 5/24 7/24 1/24 
O02 1/8 3/8 5/8 
- (22051/69, 1965/23, 6767/25)" 
= (515/1373, 25888/9137, —9428/3017)". 


x] x2 X3 x4 X5 X6 RHS 
-1/8 1 —1/6 —5/8 
1/8 -1/2 -19/8 1 
3/8 -1/6 1 7/8 
515/1373 25888/9137 —9428/3017 1 


437/482 439/1363 3118/1127 104/14021 5287/394 113/813 | —1755/2036 


Iteration 8: 
3. J = {1, 3}. 
4. 
AB = (95/100) min{ (437/482) /(515/1373), (3118/1127) /(25888/9137)} 
= (95/100)(2780/2847) = 500/539, g =3. 
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3. 
Zy = (437/482, 3118/1127, 5287/394)" 
— (500/539) (515/1373, 25888/9137, —9428/3017)" 
= (733/1312, 214/1547, 9350/573)'. 
x1 x2 X3 X4 x5 X6 RHS 
-1/8 1 —1/6 —5/8 
1/8 1/2 -19/8 1 
3/8 -1/6 1 7/8 
515/1373 25888/9137 —9428/3017 1 


733/1312 439/1363 214/1547 104/14021  9350/573 113/813 | —2450/1369 


~7/24 —5/24 -11/24 2 ~1/6 
6. ag(R) =| 5/24 7/24 1/24] | -2] =| -1/6 
1/8 3/8 5/8 0 


on 


7. R=. 
11. max{—1/6, —1/6, —1/2} < 0. 
13. Setting x3 = 9137/25888 gives 
Xp = (9137/25888)(1/6, 1/6, 1/2)" 
= (9137/155328, 9137/155328, 9137/51776)'. 


Thus, the approximate basic optimal solution and optimal value are 


® & (0, 9137/155328, 9137/25888, 9137/155328, 0, 9137/51776)', 


Ff =, 3, 1, 4)(9137/155328, 9137/25888, 9137/155328, 9137/51776)" 
= 9137/4854. 


On the other hand, the exact basic optimal solution and optimal value are 
x* = (0, 1/17, 6/17, 1/17, 0,3/17)', f* = 32/17. 


The errors are about 107°, which are accumulated from the computation of a. 
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26.3 Facial Affine Interior-Point Algorithm 


In this section, a new interior-point algorithm is described first. The search direction, 

derived via the affine scaling, is just the same as that in Dikin’s affine algorithm. 

However, the new framework allows for forming a better interior-point algorithm. 
Let x be the current interior-point. Consider the dual problem below: 


max g =b'y, 


Si (A? a) (2) = Xc,z>0. (26.3) 


Note that there is a 1-to-1 correspondence between columns of the unit matrix J and 
indices of z. 

To begin with, we perform the dual elimination by orthogonal transformations 
(Part I). Since A is of full row rank, hence XA? is of full column rank, there exists 
the QR factorization 


XAT = (QO), Qs) (7) = OR, (26.4) 


where (Q1, Q3) is orthogonal, partitioned as 0; € R”*” and Q3 « R’*"—-™), and 
R, € R”*" is nonsingular upper triangular. 
We cite from Sect. 10.2 of Part I the search direction below: 


Ax’ = PXc = (1 — XA'(AX*A')"!AX)Xc. (26.5) 
The following reveals that the search direction in x’-space, used in Dikin’s affine 
algorithm, can be obtained alternatively by using the matrix Q3. 
Proposition 26.3.1 The vector Ax' defined by (26.5) is equal to Q3 O}Xc. 


Proof Substituting (26.4) to (26.5) and noting Q| Q; = I and Q; O{ + 030} =1 
gives 


Ax! = (1 ook (RToToik:) TOT) ive (u = 2:01) f= 03 (3 Xe) 
(26.6) 
oO 


Therefore, we are led to the following variant of the affine interior-point 
algorithm. 


Algorithm 26.3.1 [Affine interior-point algorithm with QR factorization] Let A € 
(0, 1), € > 0 be given. Initial: interior-point x > 0. 
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This algorithm solves the standard LP problem. 


. Compute Ax’ by (26.6). 

. Stop if \|Ax'|| < € (optimality achieved approximately). 
. Stop if Ax’ < 0 (Unbounded below). 

. Update by X = % — 4X Ax'/ max(Ax’). 

. Go to step 1. 


MA KR WN Ee 


Since Algorithm 26.3.1 and Dikin’s algorithm are equivalent in theory, their 
computational effort involved depends on how to compute the same search direction, 
including the realization, the sparsity of A, the number n — m compared with m, etc. 
We will not go into details here because what we are really interested in is not the 
preceding Algorithm but its facial variant, as derived as follows. 

We premultiply the augmented matrix of the equality constraints of (26.3) by 
OT = [Q}, Q3]", leading to the so-called triangular form, i.e., 


Tot ee ek ae 
O'(XA rlda=(4 oF ote) (26.7) 


The preceding represents the linear system which is equivalent to the dual equality 


constraints. By Proposition 26.3.1, it is known that the south-east submatrix 
(Q3| Q3 Xc) gives the projection Ax’, defined by (26.6), i-e., 


Ax' = Q3 (Of xc) (26.8) 
Then, update x by 
R =X —AXAx'/ max(Ax’), (26.9) 
where A ¢€ (0, 1) is the stepsize. If it goes to the next iteration at this point, the 
resulting is just the same as Algorithm 26.3.1. 


On the contrary, we will carry out a series of facial iterations in x’-space, starting 
with 


and moves from face to lower-dimensional face. 
Assume that Ax, = Ax’ ¢ 0. Update the x}, by the following formula: 


Ky =e — AAx'/ max(Axp), 
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and determine an index q such that 


q =argmax {Ax |j=1,....}. 


It is not difficult to show that (XAT : e,) is of full column rank. If the QR 
factorization of it is available, then the orthogonal projection of the objective 
gradient Xc onto the null space of 


can be computed similarly as before, and it is therefore able to update the solution 
once more in x’-space. Note that the g-indexed component of the projection equals 
0, hence the g-indexed component of the solution remains unchanged. 

Assume that after performing k (< n — m) such iterations, there are index sets 


WET ..k, BSe+ 1.0.00): 


Let the QR factorization (X A™ : Ivy) = OR be available. Premultiplying by Q7 the 
augmented matrix of equality constraints of (26.3) gives 


O'(XA!: Iy : Ip|Xc) 


OQ; XA" Olly OFlg | Q}Xc Ru Riz OF Tz | Q}Xc 

=| QXA™ OFIy O31 |Q3Xc] =| O Ro OyIp | Q5Xc 

Q}XAT OFln O3lB | O}Xc 0 0 Q3Ip | O3Xc 
(26.10) 


where the orthogonal matrix Q = (Q1, Q2, Q3) is partitioned as Q; € R”™*"", 
QO. € R"** and 03 e R™"—"—®, and Ry, € R&"tFHX"+) is nonsingular upper 
triangular. The preceding is the kth triangular form, whose south-east submatrix 
gives projection 


Ax), = (Ol). (Q5Xc). (26.11) 


Assume that Ax, £ 0. Since Ax), = (OF Iy)™(Q}Xc) = 0 and x) remains 
unchanged, one just need to update 


£ = Fy — AW Ax, (26.12) 
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where 
of /- : =f / / . 
a = iy /Ax), = min {#/Ax) | Ax; > 0, je BY, (26.13) 


which is the stepsize from the current solution to the nearest boundary. It is clear 
that the new solution is again an interior point. Then move q from B to N, and go 
to the (kK + 1)th facial iteration. 

The forgoing process terminates when k = n — m or orxX c = 0, hence the Ax, 
defined by (26.11) vanishes. In this case, substituting zz = 0 to the dual equality 
constraints, represented by (26.10), leads to the upper triangular systems 


Ri Ri2 y QO} Xc 
= - |. 26.14 
( 0 (2) ( axe ( ) 
Assume that (y, Zyv) is the solution to this system. 


If zy => 0, it is not difficult to show that x is the optimal solution to the following 
problem: 


If A € (0, 1) is sufficiently close to 1, then xy can be arbitrarily close to 0. If A is 
predetermined to be close to 1, therefore, x can be regarded as an approximately 
optimal solution to the original problem, and the solution process terminates. 

In the other case when zy # 0, the facial iterations are finished. The related 
solution in the original space can be obtained by 


x= Xx, 


and anew X can be formed to be ready to carry out another series of facial iterations. 
The trick of the algorithm is that the QR factors in each facial iteration can be 
obtained recursively. In fact, the last n — m — k — 1 components of the g-indexed 
column of (26.10) can be eliminated by Givens rotations. 
Let 0 e RU-™—)x(2—m—L) be the product of Givens rotations such that 


O03eq = nel. 


Then oT =[/ oTyr ot is just the required factor for the (k + 1)th facial iteration. 
It is seen that the k and (k + 1)th triangular forms are the same, except for the 
submatrix, associated with Q3. On the other hand, at the beginning of each series 
of facial iterations, it is necessary to compute the QR factors from scratch since X 
changes. 

The overall steps can be summarized to the following algorithm. 
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Algorithm 26.3.2 (Facial Affine Interior-Point Algorithm) Given A € (0, 1). 
Initial: interior point x > 0. This algorithm solves the standard LP problem. 


. Set k = 0, and compute triangular form (26.7). 

. Compute Ax’ = 03(Q;Xc). 

. Stop if Ax’ < 0 (lower unbounded). 

. Determine w and q such that a = x,/Ax) = min{x';/Ax’ | Ax, >0, j € B}. 

. fw 4 0, update: x, = 2, — AwAx,. 

. Setk =k + 1, and update (B, N) by bringing g from B to N. 

. Go to step 9 ifk =n — mor Ol xXe=0. 

. Eliminate the (m + k + 1) to nth components of the g-indexed column of the 
triangular form by Givens rotations. 
9. Go to step 2. 

10. Solve the upper triangular system (26.14). 

11. Stop if zy > O (approximate optimality achieved). 

12. Set% = Xx’. 

13. Go to step 1. 


CSAYADNDMN PWN Ke 


The algorithm, developed by Pan (2013), is slightly different from the preceding 
algorithm, as it carries out the facial iteration using 


al = 


/ / 
Xp =Xp—adAxp 


rather than (26.12). Therefore, the resulting iterate is not interior but boundary point. 
As Algorithm 26.3.2 is close to Pan’s algorithm, we cite the latter’s computational 
results to give a clue on its performance. 

The associated computational experiments were carried out on a Pentium III 
550E PC with Windows 98 operating system, 168 MB inner storage, and about 
16 decimal precision. Visual Fortran 5.0 compiler was used. There were following 
three dense codes involved: 


1. AIP: affine interior-point Algorithm 10.2.1 (Part I). 
2. VAIP:Algorithm 26.3.1. 
3. PAIP:Algorithm 26.3.2. 


The preceding codes are tested on the 26 smallest (by m + n) Netlib standard 
problems. The first set involves 16 smaller problems, and the second set is the rest 
10 problems (Appendix C: Table C.4, problems AFIRO-DEGEN72). 

Table 26.1 below lists iterations and time ratios: 


Table 26.1 Iteration and AIP/VAIP AIP/PAIP 

ee Problem Iterations |Time | Iterations | Time 
Set 1 (16) 0.98 0.32 | 6.32 1.47 
Set 2 (10) 0.92 0.25 | 3.55 1.52 


Average (26) | 0.95 (0.26 | 4.56 1.52 
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From the bottom line of the preceding Table, it is seen that the total iteration 
and time ratios of AIP to VAIP are 0.95 and 0.26, respectively. So, the former 
outperforms the latter. However, PAIP outperforms AIP significantly: the total 
iteration and time ratios of AIP to PAIP are 4.56 and 1.52, respectively. Therefore, 
the introduction of the facial iterations is favorable. 

On the other hand, the time ratio of AIP to PAIP is much less than their iteration 
ratio (1.52 vs 4.56), since each iteration of the latter is more time-consuming due to 
the orthogonal transformations. 


Reference 


Pan P-Q (2013) An affine-scaling pivot algorithm for linear programming. Optimization 62:43 1— 
445 


Chapter 27 ® 
Decomposition Principle crests 


It has long been a challenge to solve large-scale LP problems that require a huge 
number of iterations and storage. One option for solving such a problem is to use 
some decomposition methods, like the Dantzig—Wolfe or Benders decomposition 
method. Unfortunately, applications of these methods are limited to certain types of 
structured problems. Very large problems cannot be solved at present although the 
computer hardware is advanced. 

On the reality side, however, decomposition arrangements have been put into 
effect successfully for many years. For instance, the football World Cup follows a 
“parallel decomposition” competition system: firstly preliminary selection contest is 
undertaken in each of the six continents; then the winners take part in the final round, 
and the champion stands out fi. On the other hand, an arena contest follows a “serial 
decomposition” competition system: the champion is produced through a series of 
games between a temporary winner and a challenger successively. Without a doubt, 
it is impossible to organize any big event without any decomposition scheme. 

Solving very large-scale LP problems can be closely analogized to the above 
events. This is the theme of this chapter. Firstly, we introduce two decomposition 
algorithms in the next section and then address the decomposition principle— 
“arena contest,” which the new decomposition algorithms follow. In the subsequent 
sections, the decomposition method is illustrated with instances of optimization 
problems, such as the standard LP problem, bounded-variable LP Problem, and ILP 
Problem. Practical remarks are made in the final section. 


27.1 New Decomposition Method 


Firstly, we show that the standard problem can be solved by handling two smaller 
subprograms in succession. 
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Let Bo be an initial feasible basis, associated with feasible basic solution x. 
Partition nonbasis N to two sections (V1, N2). Denote the corresponding sections 
of c by (cB, Cn, CN). The first subprogram is of the following form: 


ae | T 
min CBX Bo + CN XN» (27.1) 
s.t. Boxp, + Nixn, =b, xj =0, j € BoOUM, 
whose optimal basis (basic solution or value) is termed suboptimal with respect to 
the original problem. Based on the resulting suboptimal basis, say B,, we construct 
a second subprogram as follows: 


min Ch XB, + CN XNo» . (27.2) 
s.t. Byxg, + Noxn,=b, xj =0, j € BJ UND. 

We might as well assume that the problem is not unbounded. Then each 
suboptimal solution corresponds to a feasible basic solution to the original problem; 
e.g., after the first subprogram is solved, the components, associated with Bo U Nj, 
of the feasible solution x change to the new values of the suboptimal solution, while 
the other components remain unchanged. Note that all nonbasic components of 
the suboptimal solutions are zero, and so are those of the corresponding feasible 
solutions to the original problem. 

It will be shown that optimality of the original problem is achieved if the 
suboptimal values of the two subprograms are equal. Otherwise, take the suboptimal 
basis, obtained by solving the second subprogram, as anew Bo to repeat the previous 
steps. 

The preceding can be generalized by partitioning the nonbasis to multiple 
sections. Let Bo be the initial feasible basis, and let (Nj,..., Ns), s > 2 bea 
partition of the nonbasis. The tth subprogram (ft = 1, ..., 5) to be solved is of form 


aad T 
min cp *B-1 + Cy, xNr> 


(27.3) 
S.t. Br-1XB,_, + N;:XN, = b, Xj = 0, ye By U Ni, 


where B;— is a suboptimal basis, obtained by solving the (¢ — 1)th subprogram 
(t =2,...,5). 
This scheme can be organized into the following algorithm model. 


Algorithm 27.1.1 (Decomposition Algorithm 1) Let the integer (s > 2) be given. 
Initial: feasible basis Bo. This algorithm solves the standard LP problem. 


1. Partition A\ Bo to (Mj,..., Ns). 
2. Sett = 0. 

3. Sett =t+1. 

4 


. Solve subprogram (27.3) by the simplex algorithm (Part I): 


(1) Stop if lower unbounded. 7 
(2) Ift <s, go to step 3 with the suboptimal basis B; and objective value f;. 
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5. Stop if fy = fi. 
6. Set Bo = Bs. 
7. Go to step 1. 


Iterations associated with t = | s form a “circle” (solving the s subprograms). 
The solution process involves a series of circles, in general. 


Lemma 27.1.1 Algorithm 27.1.1 generates a sequence of feasible bases, with 
objective values decreasing (not necessarily strictly) monotonically. 


Proof We might as well assume that the algorithm terminates at the end of the 
second circle. 

Let us take a look at the first circle. As the initial basis Bo is feasible to the 
original problem, the suboptimal bases B; are well-defined (t = 1, ..., 5). Itis clear 
that each basis B;, and associated objective value f; are not only optimal to the fth 
subprogram but also feasible to the original problem. Since each f; is optimal to 
the tth subprogram and feasible to the (t + 1)th subprogram, the objective values 
decrease (not necessarily strictly) monotonically. Likewise, it can be shown that the 
objective values decrease monotonically within the second circle; therefore, so do 
all the objective values since Bo for the second circle is just the end suboptimal basis 
B, of the first circle. oO 


Theorem 27.1.2 Under the assumption of nondegeneracy, Algorithm 27.1.1 termi- 
nates either at 


(i) Step 4(1), detecting lower unboundedness of the original problem (12.1), or at 
(ii) Step 5, generating an optimal basis to the original problem 


Proof Assume at the moment that the algorithm terminates. By Lemma 27.1.1, the 
condition f, = fi implies that all the feasible values are equal. 

It is clear that the termination at step 4(1) implies lower unboundedness of the 
original problem. Assume that the process terminates at step 5. We show (ii) by 
induction on the number s of sections. 

Consider the case s = 1. Since By is an optimal basis to the first subprogram, the 
corresponding optimal dual solution, say (7, Z), satisfies 


Z BUN) = CBN, — (BOUN)'¥ 20, F = By ep, (27.4) 

As A = {Bo, Ni, N2}, so we only need to show that 
Zn, = en, — NSP >. (O75) 
Since fo = f1, B, is also optimal to the second subprogram. If (27.5) does not hold, 
then it follows from xg, = B, ‘b> 0 (nondegeneracy assumption) that the By, 
is not optimal to the second subprogram, a contradiction. Therefore, (27.5) holds, 


which together with (27.4) implies together that B; is an optimal basis to the original 
problem. 
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Assuming that (i1) is valid for l<s<t with t2 1, we will show the validity 
fors =t+1.Since fi41 = fi, it holds that f, = f; (Lemma 27.1.1). According to 
the assumption of induction, B, is therefore an optimal basis to the problem 


min Chih + Co dts (27.6) 
s.t. Boxy + N'xy =b, xj >0, 7 € BUN’, , 
where N’ = N; UN2U...UN;. Taking a look at the partition (Bo, N’, N;+1) of the 
original problem, and noting that the statement is valid for s = 2, it can be asserted 
that B is an optimal basis to the original problem. 

Let us turn to the termination of Algorithm 27.1.1. All associated subprograms 
are feasible, and hence the simplex algorithm utilized to solve them terminates 
under the assumption of nondegeneracy. Thus, it does not terminate only if there 
are infinitely many circles, and f, < f| holds for each. This implies that there 
are infinitely many bases, as leads to a contradiction. Therefore, Algorithm 27.1.1 
terminates. Oo 


The shortcoming of subprogram (27.1) (or (27.3)) is that the entering set Ny (or 
N;) is predetermined without considering the associated reduced costs at all. This 
may be remedied as follows. 

Let € be the optimality tolerance and let a real number (o0 > € > O) and an 
integer, (m’ < m) be given. Denote Bo the current feasible basis and by N the 
candidate indices from which to choose the entering set Nj. Initially, just take the 
nonbasis as N, i.e., N = A\Bo. Assume that Zy is the nonbasic reduced costs. 
Then, Nj is determined by 


M={j EN |Z; < =o}, (27.7) 


subject to |Ni| < m’. So the number of indices in Nj is no more than m. If |Ni| > 0, 
subprogram (27.1) is solved. Then we set N = A\(BoUN7) for the next iteration. In 
the solution process, o is decreased by a percentage t to maintain |; | reasonably 
large whenever |Nj| < m’. Assuming |N\| = 0, optimality of the original problem 
is achieved if o = €, or Nj is determined by (27.7) again with a reduced o if > e, 
and so on. 

The preceding discussions lead to a variant of Algorithm 27.1.1, formulated as 
follows. 


Algorithm 27.1.2 (Decomposition Algorithm 2) Give reals € and t such that 0 < 
e < 1,0 < t < 1 and integer m’ < m. Initial: 0 > €; feasible basis Bo and 
nonbasis NV. This algorithm solves the standard LP problem. 


. Compute Zy = cy — N'y, Bly = CBo: 
. Determine set Nj = {j € N|z; < —o}. 
. If |Ni| < m’, set o = max{e, to}. 

. Go to step 7 if |Nj| > 0. 

. Stop if o = € (optimality achieved). 
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Go to step 2. 
Solve subprogram (27.1) by the simplex Algorithm 3.1.1 (Part D: 


(1) Stop if lower unbounded. 
(2) Set N = A\(Bo U Nj), and then set the yielding suboptimal basis to Bo. 


Go to step 1. 


27.2 Decomposition Principle: ‘“‘Arena Contest’ 


The preceding decomposition algorithms work as if some “arena contest’ is 
organized to generate the “team champion” via a series of “competitions.” 


The number of players (columns) in the whole area (coefficient matrix) is too 


huge to match in the limited sports facilities (available hardware) to produce a team 
champion (optimal basis). Therefore, a system is formulated and executed to govern 
the competition (optimization), so that a series of relatively small games can go 


on 


smoothly in the limited facilities. The special idea behind each decomposition 


algorithm may be explained as follows. 


(A) Algorithm 27.1.1: 


(B 


At the beginning, an initial team (basis Bo) is selected from players (columns) 
of the area (coefficient matrix A) under some criterion. The other players 
(nonbasic columns JN) are divided into s teams (Nj,..., Ns). The first match 
takes place in the facilities among the initial team and the first team (Bo, 1), 
producing a first winner team (suboptimal basis B,). Then, the second match 
takes place among the first winner and the second team (B,, N2), and so 
on. Once the sth match is finished, the last winner team (suboptimal basis 
Bs) is declared as the ultimate champion (“‘optimal” basis) if it has the same 
record (suboptimal value) as the first winner; otherwise, the next round of 
competitions is carried out by taking the last winner as a new initial team 
(setting Bo = B;). 

Algorithm 27.1.2: 

After a temporary winner team (Bo) is selected, a challenging team (J) is 
determined in a tryout among all the other players (V) under a criterion, which 
will be relaxed whenever the member of the challenging team is less than some 
threshold. Then a match takes place between the temporary winner team and 
the challenging team (Bo, N1) producing a new temporary winner team (setting 
the last suboptimal basis to Bo), and repeat. If no player wins out in the tryout 
under the criterion of the lowest standard (zy > —e), the last winner team; 
i.e., the end suboptimal basis, is declared as the ultimate team champion (the 
optimal basis). 


Ym 
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27.3 Illustration on Standard LP Problem 


In this subsection, an instance is offered for illustration of the decomposition 
principle on a standard problem. 


Example 27.3.1 Solve the following LP problem by Algorithm 27.1.1 with s = 2: 


min f = —2x, + x2 + 3x3 — 5x4 — 3x6 — 7x7 — 4xg — 2x9 


S.t. —3x2 — 2x3 + x4 + 4x7 + —2x9 = 1 
3x1 + 4x3 + X5 +2x7+ x9 +5x9 =2 
—xX, + x2 4+ 23 + x6 + 8x7 — 6xg + 3x9 = 8 


xj =0, f=1,...,9. 


Answer Initial feasible basis and nonbasis: Bo = {4, 5, 6}, N = {1, 2, 3, 7, 8, 9}. 
Outer iteration 1: 
Subiteration 1: 


1. Ny = {1, 2,3}, No = {7, 8, 9}. 

2.t=0. 

3.¢=041=1. 

4. Call Algorithm 3.1.1 (Part I) to solve the first subprogram (Bo = {4,5, 6}, Ni = 
{1,.2,.3}) 

min —2x; + x2 +3x3-—5x4 — 3x6 
s.t. —3x2 — 2x3 + x4 =1 
3x1 + 4x3 + x5 =) 
xX +x2+ x3 +xp =8 
xj 20, j=1,...,6. 

6. The resulting optima basis: Bo =  {1,2,4}, associated with xg, = 
(2/3, 26/3, 27)' with objective value fj = (—2, 1, —5)(2/3, 26/3,27)' = 
—383/3. 
t=l<s. 


Subiteration 2: 


3.t=1+1=2. 
4. Call Algorithm 3.3.1 (Part I) to solve the second subprogram (By = 
{1, 2,4}, No = {7, 8, 9}) 


min —2x, + x2 — 5x4 — 7x7 — 4x8 — 2x9 
S.t. —3x2 + x4 + 4x7 —2x9 =1 
3x1 +2x7+ xg +5x9 =2 
—x, + x2 + 8x7 —6xg+3x9 =8 
xj >0, j =1,2,4,7, 8,9. 
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6. The resulting optima basis: Bj = {2, 4, 8}, associated with xg, = (20, 61, 2° 
with fo = (1, —5, —4)(20, 61, 2) = —293 
t=s 

7. fro # fi. 

8. Bo = {2,4, 8}, N = {1, 3,5, 6, 7, 9}. 

Outer iteration 2: Set Bp = By = {2, 4, 8}. 

Subiteration 1: 


1. Ny = {1,3,5}, No = {6, 7, 9}. 
2.t=0. 
3.f=0+1=1. 
4. Call the simplex Algorithm 3.3.1 (Part I) to solve the first subprogram 
min —2x; + x2 +3x3—5x4 —4xg 
S.t. —3x2 — 2x3 + x4 =1 
3x] + 4x3 +x5+ xg =2 
—x1 + Xo + x3 — 6xg = 8 


xj >=0, j =1,2,3,4,5,8. 


6. The resulting suboptimal basis is Bo = {2,4, 8}, associated with xg, = 
(20, 61, 2)? with fi = —293. 
t<-s. 


Subiteration 2: 


3,.t=14+1=2. 
4. Call Algorithm 3.3.1 (Part I) to solve the second subprogram (Bp = 
{2, 4, 8}, No = {6, 7, 9}) 


min X2 5x4 3x6 7X7 Axe 2x9 


s.t. —3x2 + x4 + 4x7 —2x9 =1 
2x7 + xg +5x9 = 2 
x2 + x6 + 8x7 — 6xg + 3x9 = 8 
xj >0, j =2,4,6,7, 8,9. 
6. The resulting suboptimal basis is Bj} = {2,4, 8}, associated with xg, = 
(20, 61, 2)" with fo = (1, —5, —4)(20, 61, 2)? = —293 
t=s 


7. f2 = f\. The optimal solution and optimal value are 


x = (0, 20,0,61,0,0,0,2,0)', f= —293, 


which can be verified by solving the original problem as a whole by the simplex 
algorithm. 
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27.4 Illustration on Bounded-Variable LP Problem 


Algorithm 27.1.1 can be modified to solve the bounded-variable problem if the 
simplex algorithm called from step 4 is replaced by a bounded-variable problem 
solver. 


Example 27.4.1 Solve the following bounded-variable problem by a modified 
algorithm, Algorithm 27.1.1, with s = 2, using Algorithm 8.1.1 (Part I) as the 
subalgorithm: 


min f = 2x; — x2 + 3x3 — 6x4 


s.t. —2x, + 3x2 — 4x3 4+ 2x4 + x5 = 14 
—3x, + 4x2 — 5x3 + 6x4 + x6 = 16 
Xx, — 2x2 + 2x3 — 7x4 + x7 =-15 


—-15 <x, <30, -—12<x.<20, —17< x3 < 10, 
—8 <x4< 15, -—200<x5<26, —200< x6 < 34, 
0 < x7 < 200. 


Answer Initial feasible basis and nonbasis: Bo = {5, 6,7}, N = {1, 2, 3, 4}. 
Outer iteration 1: 
Subiteration 1: 


1. Ny = {1,2}, No = {3,4}. 
2.t=0. 
3.f=04+1=1. 
4. Construct the first subalgorithm with Bp = {5,6,7}, N; = {1,2} by setting 
Xo = (-17, 15)", where the right-hand side is changed to 
14 —42 —84 


b—Noin,= | 16]—-| —56] = ey = | —159 ], that is, 
-15 2-7 124 
min 2x1 — x2 
s.t. —2x, + 3x2 + x5 = —84 
—3x, + 4x2 + x6 = —159 
Xx, — 2x2 + Xx7 = 124 


-15 <x; < 30, -—-12<x2<20, —200< x5 < 26, 
—200 < x6 < 34, O< x7 < 200. 


Call Algorithm 3.1.1 (Part I) to solve the preceding. 
6. The resulting suboptimal basis is Bp = {2,5,7}, associated with the basic 
solution and objective value 


x = (—15, —1, —17, 15 — 111, —200, 137), fi = (2, -1, 3, —6, 0, 0, 0)x = —170. 
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t=l1l<s. 
Subiteration 2: 


3.f=1+1=2. 
4. Construct the second subalgorithm with Bp = {2,5, 7}, N2 = {3, 4} by setting 
(x1, x6) = (—15, —200), where the right-hand side is changed to 


14 —20 15 —16 
b— N2xn, = 16]}]-—]|-31 ]= es = 171 }, 
-15 10 0 
yielding 
min —x2 + 3x3 — 6x4 
S.t. 3x2 — 4x3 + 2x4 + x5 = —16 
Axo — 5x3 + 6x4 = 171 


—2x2 + 2x3 — 7x4 + x7 = 
-12 <x. <20, -17< x3 < 10, 
—8<x4< 15, -—200<x5< 26, O< x7 < 200. 


Call Algorithm 3.1.1 (Part I) to solve the preceding. 

6. The resulting suboptimal basis B} = {2,5, 7} is equal to Bo. 
t=S. 

7. The optimal solution and optimal value are therefore 


¥ = (-15, —1, -17, 15-111, —200, 137)", fo = (2, -1, 3, —6, 0, 0, 0)X = —170, 


which can be verified by solving the original problem as a whole by Algo- 
rithm 8.1.1 (Part I). 


27.5 Illustration on ILP Problem 


It is possible to handle large-scale ILP problems by some variant of Algo- 
rithm 27.1.1 if there is an optimal solution to the initial ILP subprogram. In 
general, however, this approach may not produce an optimal solution to the ILP 
problem, but an approximate one only. Starting from such a point, however, 
methods presented in Chap.11, in particular, the realization presented in the 
next section, are applicable for reaching an exact optimal solution. Besides, it is 
not difficult to extend the approach to solving the large-scale mixed-integer LP 
problems. 

To start-, one would solve the associated relaxed LP problem first and then 
construct an initial subprogram, based on the resulting optimal basis, if any. But 
we will do differently with the following instance. 
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Example 27.5.1 Solve the following integer LP problem by a modified algorithm, 
Algorithm 27.1.1, with s = 2, supported by some solver: 


min f = x1 — 3x2 + 2x3 — 5x4 +4%5 


s.t. 2x, + x2 — x3 + 3x4 — 2x5 4+ x6 =7 
4x1) — 3x2 — 5x4 — 4x5 + x7 = 2 
—3x, + 2x2 +43 +2x4+ x5 +xg=8 
xj 20, j =1,..., 8 integers. 


Answer Initial feasible basis and nonbasis: Bo = {6, 7, 8}, N = {1, 2, 3, 4, 5}. 
Outer iteration 1: 
Subiteration 1: 


1. Ny = {1,2}, No = {3, 4, 5}. 

2.¢t=0. 

3.f=04+1=1. 

4. Solve the first subalgorithm with Bp = {6, 7, 8}, Ni = {1, 2}, formed by setting 


XN, = (0, 0, 0): 


min x; — 3x2 


s.t. 2x, + x2 + x6 = 
4x1) — 3x2 + x7 = 
—3x, +2x2 + xg = 8 


xj =0, j = 1,2, 6, 7, 8 integers. 


6. The resulting suboptimal basis is Bp = {1, 2, 7}, associated with the solution and 
objective value 


# = (1,5,0,0,0,0,13,1)', fi =(1,—3,0,0,0,0,0,0)% = -14, 


t=l<s. 
Subiteration 2: 


3.f=1+1=2. 
4. Construct the second subalgorithm with Bp = {1, 2,7}, No = {3, 4, 5} by setting 
xn, = (0, iF, where the right-hand side is changed to 


7 10 0 7 
b— N2xn, = | 2 | — | 00 =) 2 I; 
8 01 7 
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yielding 


min x; — 3x2 + 2x3 —5x4+ x5 


s.t. 2x, + x2 — x3 + 3x4 — 2x5 =7 
4x, —3x2 —5x4—4x5+4+ x7 =—2 
—3x, +2x2 +434 2x44 x5 =7 


xj >0, j =1,2,3,4,5, 7 integers. 
Solve the preceding by a code. 


6. The resulting suboptimal basis is B; = {4, 5, 7}. 
7. The optimal solution and optimal value of the subprogram are 


x2 = (0,0,0,3,1,0,21,)7, fo =(1,-3,2,—5, 1,0, 0, ox = -14, 


t=S. 
7. fo = fi = —14. x or x would be an acceptable solution to the original 
problem. 


The exact optimal solution to the original problem is 


x* = (3,0,0,5,7,0,43,0)7 f*=-15. 


27.6 Practical Remarks 


Many large-scale LP problems involve a large number of variables, compared with 
the number of constraint equations (m <« n). In this case, it is reasonable to form the 
subprograms involving columns fewer than m or less. Moreover, in the case when 
n—m <™m, the problem can be converted by the dual elimination (Part I) to an 
(n — m) x n standard LP problem, so that a large-scale problem can be handled by 
solving small programs with the decomposition algorithm. 

The partition (N1,..., Ns) involved in Algorithm 27.1.1 (or Algorithm 27.1.2) 
is not unique. Of course, we wish that an optimal basis is contained in the first 
few sections, e.g., in the Nj section ideally. How to achieve this is worth further 
investigating. It seems promising to give the nonbasic indices an order by changing 
the pivoting-index (Part I) by taking sparsity into account, so that sparser columns 
are in front of the line to have priority to enter Nj. 

On the other hand, both Algorithms 27.1.1 and 27.1.2 require an initial feasible 
basis to get started, and hence a two-phase procedure is needed to solve general 
LP problems. Although any existing Phase-I approach applies for this purpose, 
it is preferable to deal with some Phase-I auxiliary problem by following the 
decomposition principle. 

It should be pointed out that the feasibility of the initial basis is not necessary. It 
suffices to ensure that the initial subprogram is itself feasible. 
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The potential use of the decomposition principle would be of great importance. 
In fact, Algorithms 27.1.1 and 27.1.2 could be generalized along the following two 
lines: 


(A) Instead of the simplex algorithm, other basis-based algorithms, e.g., the dual 
simplex algorithm or the reduced simplex algorithm can be used as the 
subalgorithm, perhaps with slight modifications. Even the face algorithm, 
producing an optimal face, applies. But primal algorithms seem to be amenable, 
compared with dual algorithms. 

(B) Some hard optimization problems could be handled if they have linear con- 
straints and separable variables; e.g., large integer or mixed-integer LP prob- 
lems would be approximately solved by the algorithms, supported by the 
branch-bound or cutting-plane algorithm amenable to small such problems. 
Further investigation is expected in this aspect. 


Algorithms 27.1.1 and 27.1.2 are designed for solving standard LP problems, 
where all nonbasic components of solutions are of default value 0, and hence may 
be omitted in the solution process. For other problems, such as the bounded-variable 
problem, a feasible basic solution should be available initially. In each iteration, the 
components, associated with the current subprogram, of the solution are updated, 
while the other components remain unchanged. For integer or mixed LP problems, 
an initial feasible integer solution is certainly preferable, but not necessary. A 
feasible solution to the associated relaxed LP problem can be taken as an initial 
solution to get some variant of Algorithm 27.1.1 or 27.1.2 started, though there is a 
need for the existence of an optimal solution to the initial integer subprogram. 


Appendix A 
On the Birth of LP and More 


It would be interesting, inspiring, and beneficial to review the birth of LP. 

LP can be traced back to as early as 1820s. French mathematician J.B.J. Fourier 
(well-known due to the mathematical series named after him) in 1823 and Belgian 
mathematician V. Poussin in 1911 each wrote a paper related to LP. Unfortunately, 
these isolated efforts went unnoticed. In 1939, the academician L.V. Kantorovich 
(1960) of former Soviet Union academy of sciences published a book entitled 
Mathematical Method of Production Management and Planning, concerning LP 
models and their solution, but it was unfortunately neglected by the authority and 
had not been known by the academic communities. In 1941, FL. Hitchcock’s 
original paper on the transportation problem once again did not draw any attention, 
until the late 1940s and early 1950s when it was finally rediscovered after the 
simplex method came around. 

Indeed, human activities are the only source of all scientific theories and method- 
ologies. Although the World War II led to devastating losses and casualties, the war 
efforts thrust science and technology forward and hastened emergence of many new 
branches of learning. In particular, Operations Research and Optimization are one 
area that has greatly benefited from the wartime demand. 

After receiving his Ph.D., George B. Dantzig became a mathematical advisor 
to the US (Cottle et al. 2007) Air Force Controller in 1946. He was challenged to 
mechanize the planning process such that a time-staged deployment, training, and 
logistical supply program can be more rapidly computed using desk calculators. 
Using an active analysis approach, he formulated a model without an objective 
function and tried to solve it. Dantzig recalled the situation at that time at the XI- 
th International Symposium on Mathematical Programming (held at the University 
of Bonn, W. Germany, from August 23 to 27 in 1982), by giving the following 
example: 


How to assign 70 men to 70 jobs ? 
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It is required that each job must be filled and each man must be assigned. 
Therefore, one needs to find a good “activity,” an 1-to-1 correspondence between the 
men and the jobs. There are 70! > 10! activities. It is impracticable to determine 
the best among them by comparing all, because the number is too large even though 
being finite. Dantzig said (1982, Bonn): 


Suppose we had an IBM 370-168 available at the time of the Big Bang 15 billion years ago. 
Would it have been able to look at all the 70! combinations by the year 1981? No! Suppose 
instead it could examine | billion assignments per second? The answer is still no. Even if 
the Earth were filled with such computers all working in parallel, the answer would still 
be no. If, however, there were 10“ suns all filled with nano-second speed computers all 
programmed in parallel from the time of the big bang until sum grows cold, then perhaps 
the answer is yes. 


This example illustrates the ordeals faced by decision makers before 1947. 
Therefore, “experience,” “mature judgement,” and “rules-of-thumb” had to be relied 
on to come up with an “acceptable” scheme, which was often far from the optimal. 

In the summer of 1947, Dantzig founded the simplex method, marking the birth 
of LP. The method’s prefix simplex came from a chat between G.B. Dantzig and 
T. Motzkin (Dantzig 1991) in the early days of LP. The latter indicated that the m 
columns of the basis matrix and the entering column just form a simplex in the m- 
dimensional space. Therefore, each iteration in Dantzig’s method can be viewed as 
a movement from a simplex to another. Dantzig accepted his suggestion by taking 
simplex as the prefix. 

On October 3 of the year, Dantzig visited one of the greatest scientists of the 
twentieth century, J. von Neumann, and talked about his results. The latter imme- 
diately got the basic idea behind the method and indicated potential relationship 
between the method and game theory which he was working on. 

In 1948, Dantzig attended a conference of the Econometric Society in Wis- 
consin held in Wisconsin, meeting well-known statisticians, mathematicians, and 
economists, such as H. Hotelling, T. Koopmans, J. von Newmann, and many others 
who became famous later but just had begun their career then. After young unknown 
Dantzig presented his simplex method and LP, the Chairman called for discussion. 
There was silence for a moment. Then, Hotelling said, “But we all know the 
world is nonlinear.’ When Dantzig was frantically trying to properly reply, Von 
Neumann came to his rescue, “The speaker titled his talk ‘Linear Programming’ 
and he carefully stated his axioms. If you have an application that satisfies the 
axioms, use it. If it does not, then don’t.” Of course, what Hotelling said is right, the 
world is highly nonlinear; but, fortunately, the vast majority of nonlinear relations 
encountered in practice can be approximated by linear ones. As we all know, the 
subsequent development completely exceeded people’s imagination, even Dante’s 
own imagination. 

The implementation of the simplex method began at National Bureau of Stan- 
dards in the USA. Around 1952, a team led by A. Hoffman of National Bureau 
of Standards tested the simplex method and compared with T. Motzkin’s popular 
method at that time. Unambiguously, the former defeated the latter. In 1953- 
1954, W. Orchard-Hay (1954a started his pioneer work. He worked out the first 
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simplex-method-based commercial software and performed on a primitive computer 
by today’s standard. Subsequently, his implementation techniques were used and 
enhanced by many scholars. As a result, the simplex method became a powerful 
practical tool. If the simplex method is used, the assignment problem mentioned 
above can be solved in | s on IBM370-168, which is not even the fastest computer 
at that time when Dantzig told his story. 

A number of Nobel Economic Prizes awarded in the past are related to LP. 
Russian-American economist Wassily Leontief took the Nobel Economic Prize in 
1973 for developing the input-output model, representing the interdependencies 
between different branches of a national economy or different regional economies. 
Kantorovich and Koopmans shared Nobel Economic prize in 1975 for their 
contribution of resources allocation theory based on LP. The award was also given 
to Professors K. Arrow, P. Samuelson, H. Simon, and L. Herwricz et al., several 
decades later when they paid close attention to LP during the early days of their 
professional career. On the practice side, applications of LP and the simplex method 
have brought about enormous economic and social benefits in the past. The simplex 
method was selected as one of the “Top Ten Algorithms of the Century.” Highlighted 
in the January/February 2000 issue of Computing in Science & Engineering, a joint 
publication of the American Institute of Physics and the IEEE Computer Society, the 
ten algorithms were regarded as ones with the greatest influence on the development 
and practice of science and engineering in the twentieth century. Again and again, 
history confirms that it is practice that spawns theories and methods that are deep- 
rooted and exuberant. 

Nevertheless, the simplex method has exponential time computational complex- 
ity (Klee and Minty 1972), while a “good” method is thought to be polynomial time. 
In fact, even finiteness of the simplex method is not guaranteed; in other words, the 
process may not terminate forever (Beale 1955; Hoffman 1953). On the other hand, 
a breakthrough was made by a former Soviet Union mathematician L.G. Khachiyan 
(1979) who proposed the first polynomial-time LP solver, the ellipsoid algorithm. 
Unfortunately, it performed poorly and is not competitive with the simplex method. 
The so-called polynomial-time complexity is only the worst-case complexity, and 
the case hardly happens in practice. 

Soon, Indian mathematician N. Karmarkar (1984) published an interior-point 
algorithm for solving a special LP model. It is not only of lower order of polynomial- 
time complexity than the ellipsoid algorithm but also appeared promisingly fast 
in practice. The algorithm initiated an upsurge of interior-point method. Many 
researchers made contributions to this area. Some implementations of it are so 
efficient that many persons thought the interior-point method to be superior to the 
simplex method for solving large-scale sparse LP problems. 

Unlike simplex methods, however, interior-point methods cannot be “warmly” 
started and hence not applicable for solving integer linear programming problems. 
Since the interior-point method cannot produce an optimal basic solution, moreover, 
there is a need for an additional purification process if a vertex solution is required. 

Meanwhile, the art of the simplex method did not stand still, but rather, was 
developing and improving. For example, due to his outstanding work on software. 
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M. Sounders won the first Beale—Orchard-Hays Prize in 1985. After P.M.J. Harris 
(1973) tested some approximate steepest-edge pivot rules; J.J.H. Forrest and D. 
Goldfarb (1992) introduced several variants of the steepest-edge pivot rule and 
associated recurrence formulas and reported very favorable numerical results. Thus, 
the simplex method and interior-point method were neck-and-neck in competition. 

In fact, the worst-case complexity is not appropriate from a practical point 
of view. The average time complexity might be more relevant. K.-H. Borgwardt 
(1982a,b) was able to show that on average the simplex method has a polynomial- 
time complexity for a certain probabilistic model, if using a certain pivot rule. 
S. Smale (1983a,b) offered a similar result. These coincide with the good perfor- 
mance of the simplex method. 

Essentially, the evaluation of a method is a practical issue, and rigidly adhering 
to theory is not wise. The merit of a method, its efficiency, precision, and reliability 
and/or stability is ultimately determined by its performance in practice. In this 
respect, the method’s finiteness or complexity is even unintentionally misleading. 
In fact, the finite/polynomial-time algorithms usually perform far worse than 
infinite/non-polynomial time ones overall, and the latter rather than the former 
plays a major role in applications. This is why this book draws its material from a 
practical point of view, focusing on effective theories, methods, and implementation 
techniques closely related to LP computations and applications. 


Appendix B 
MPS File 


MPS (Mathematical Programming System)is a kind of data file format, used to 
represent LP problems and mixed-integer LP problems. It is widely applied in 
academic as well as commercial optimization codes. 

There exist several expansions of MPS format. This appendix presents the MPS 
format based on that used in MINOS (Murtagh and Saunders 1998). Nevertheless, 
we are only concerned here with the part of liner programming only, though MINOS 
is also used to handle nonlinear programming problems. 

In the MPS format, data of the problem to be solved have to appear in specific 
columns. A MPS file is sectioned under several items as follows: 


NAME 
ROWS 


COLUMN S 

RHS 

RANGES (optional) 
BOUNDS (optional) 


ENDDATA 
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Each item must start from the first column. Symbols “.” under an item are of the 
following form: 


Column 2-3 5-12 15-22 25-36 40-47 50-61 
Content Key NameO Namel Valuel Name2 Value2 


where note rows can be inserted, with symbol “*” at the first column, and any 
characters at the 2—22th columns. 
1. NAME of the problem 


Example B.0.1 NAME row 


The name row is usually the first row of a MPS file. 

Characters NAME are in the 1-4th columns. The name of the problem is filled 
in the 15—22th columns, which can be | to 8 any characters or blank spaces. The 
name may also be used to indicate outputs, at the first row of each output basis. 

2. ROWS Section 
Example B.0.2, ROWS 


2 5.....12 
ROWS 

E ROW4 
G ROWI1 
L ROW7 
N COST 


This section lists given names of all (linear) constraints (e.g., ROW4,ROW1, 
ROW7, etc.), one row for each constraint, where a single character under Key gives 
the types of constraints, placed in columns 2—3. Name0 gives a name of 8 characters, 
placed in columns 5-12. The types are 


Key Row type 


Z2Z2T0°Q 
IA IV 


BMPS File 


Types of row EG, 


3. COLUMNS Section 


Example B.0.3 COLUMNS section 


691 


and L, respectively, indicate the constraints are 
At V AMA RALIEA “equal to,” “greater than or equal to,” and “less than or 
equal to,” and N indicates “free” or without restriction. 


1 Sie? 15.0522. 25e36%.536 A047. SOs asess 61 
COLUMNS 

Xl ROW2 2.3 ROW3 —2.4 

Xi ROW1 —6.7 ROW6 = 5.22222 

Xl ROW7 15.88888 

X2 ROW2 1.0 ROW4 —-4.1 

X2 ROWS 2.6666666 


For each variable x ;, COLUMNS section gives names and lists nonzeros a; ; of 
the corresponding columns of the coefficient matrix. Nonzero components of the 
first column must be listed before those of the second column, and so on. In case 
when there are multiple nonzeros in a column, these components are allowed to 
appear in any order. 

Usually, Key is blank (except for comment line), Name0 is the name of a 
column and Namel, Valuel give the name of a row and the nonzero value of some 
component of the column. The name of another row of the same column and value 
can be listed in Name2, Value2 of the same line, or put in the next row. Namel or 
Namez2 can be blank. 

In the preceding example, the variable named by X1 has 5 nonzero components 
located, respectively, at 

rows ROW2, ROW3, ROW1, ROW6, ROW7. 

4. RHS Section 


Example B.0.4 RHS section 


1 Sided? 191322" Do resccics 36 40....47 50........ 61 
RHS 
RHS1 ROW! —3.5 ROW3 1.1111 
RHS1 ROW4 11.33333 ROW7 —2.4 
RHS1 ROW2 2.6 
RHS2 ROW4 17.3 ROW2 —5.6 
RHS2 ROW1 1.9 


This section lists nonzeros of the right-hand side of the constraint system, with 
the same format as the COLUMN section. If components of the right-hand side 
are all zero, then only line RHS appears in the section. This section may include 
multiple right-hand sides. But only the first will be used unless indicated otherwise 
in the SPECS file. 
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5. RANGES Section (optional) 
Example B.0.5 RANGES section 


1 


Devs. se 


ROWS 


E 


mmqQ 


ROW2 
ROW4 
ROWS 
ROWI1 


COLUMNS 


RHS 
RHS1 


RANGES 


RANGE1 
RANGE2 ROW2 


12 


ROWS 


ROWI 


BMPS File 


W522? 25: enkagin 36 40....47 50........ 61 
2.0 ROW3 2.0 
3.0 ROW3 = 4.5 


1.4 ROWS 6.7 


RANGES section expresses constraints of form 


l<a 


Be a 


where / and u both are finite values. The range of the constraint is r = u — 1. This 
section only gives r, not / or u. The corresponding / or u will be determined by the 
row type of the constraint and the sign of r: 


Type of row 


E 


E 
G 
L 


Sign of r 


Lower bound/ | upper bound u 
b b+ r| 


b—|r| 
b b4 
b—Ir| 


b 
t |r| 
b 


The format is the same as the COLUMN section, where Name0 gives the name of 


a range. 
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6. BOUNDS Section (optional) 
Example B.0.6 BOUNDS section 


1 5.5212 5. sec22% 25.0288 a6 36 


BOUNDS 

UP BOUNDI XI 10.5 
UP BOUNDI1 X2 6.0 
LO BOUND! X3 —5.0 
UP BOUND1 X3 4.5 


Default bounds of all variables x ; (except for slack variables) are 0 < x; < oo. If 
necessary, the default values 0 and oo can be modified to / < x; < u via LOWER 
and UPPER in the SPECS file. 

In this section, Key gives the type of bounds ( UP and LO, respectively, indicate 
upper and lower bounds). Name0 gives the name of the bound set, and Namel and 
Valuel list the name of column and the value of bound, respectively. 


Example B.0.7 For the model of human arrangement problem 


min f =0.1x2+ 0.2x3 + 0.3x4 + 0.8x5 + 0.9x6 


s.t. xX] +x6>7 
x1 + x2 > 15 

x2 + x3 > 25 

x3 4+.x4 > 20 


X4 + X5 > 30 
x5+x%6>7 


xj 20, j=1,...,6, 
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The MPS file is as follows: 


| een Pee 2 
NAME 
ROWS 
G TEAM1 
G TEAM2 
G TEAM3 
G TEAM4 
G TEAMS 
G TEAM6 
N HANDS 
COLUMNS 

Xl 

X2 

X2 

X3 

X3 

x4 

x4 

X5 

X5 

X6 

X6 
RHS 

RHS1 

RHS1 

RHS1 
ENDDATA 


156522 
ASSIGN. 


TEAM1 
TEAM2 
HANDS 
TEAM3 
HANDS 
TEAM4 
HANDS 
TEAMS 
HANDS 
TEAM1 
HANDS 


TEAM1 
TEAM3 
TEAMS 


25....36 40....47 


1.0 
1.0 
0.1 
1.0 
0.2 
1.0 
0.3 
1.0 
0.8 
1.0 
0.9 


7.0 
25.0 
30.0 


TEAM2 
TEAM3 


TEAM4 


TEAMS 


TEAM6 


TEAM6 


TEAM2 
TEAM4 
TEAM6 


50... .61 


1.0 
1.0 


1.0 


1.0 


1.0 


1.0 


15.0 


20.0 
7.0 
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Appendix C 
Test LP Problems 


In this appendix, test problems (see Gay 1985) are listed according to increasing 
m +n order (Tables C.1, C.2, C.3, C.4). 
Source: 


NETLIB (http://www.netlib.org/Ip/data) 
Kennington (http://www-fp.mcs.anl.gov/otc/Guide/TestProblems/LPtest/) 
BPMPD (http://www.sztaki.hu/meszaros/bpmpd/). 


Notations in the tables: 


m: the number of rows of the coefficient matrix. 

n: the number of columns of the coefficient matrix. 

Nonzeros: the number of nonzeros of the coefficient matrix. 

BR: indicates whether there are BOUNDS and RANGES sections. 
Optimal value: optimal values given by the file. 
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Table C.1 Netlib problems 


No. [Name |m+n_|m___|n__| Nonzeros | BR__| Optimal value 

| (AFRO | 60 | 28 | 32 | 88 |_| ~4.6075314286E+02 
2 {Kez | 85 | at | at | 291 |B | =1.74990012998+03 
3 | SC50B 51 | 48° «| «119 ~~ | ~~ | ~7.0000000000E+01 
4 | SC5OA 131 | ~~ | -6.4575077059E+01 
5 |ADLITTLE | 154 97 | 465 | ~— ‘| 2.2549496316E+05 
6 
a 
8 
9 


Nn 


a 
oo 


Nn 
~— 


BLEND 158 521 ~~ |= | —3,0812149846E+01 
SHARE2B 176 730 | ~~ | —4.1573224074E+02 
SC105 209 281 | ~——s | -5.2202061212E+01 
STOCFOR1 | 229 474 ~~ | ~~ | -4.1131976219E+04 
10 | SCAGR7 270 130 —553°~—Cs«|~Ss| ~2.3313897524E+06 


lion 
Re |O}s1) 0 
rR} wW) Ow 


me 
oN 
oO 


11 | RECIPE 272,——«| «92)—«| 180 «| 752,~—s«| Bs | ~2.6661600000E+02 
12 | BOEING2 310 1339 BR | —3.1501872802E+02 
13. | ISRAEL 317 175 |142 | 2358 | -8.9664482186E+05 


14 SHAREIB 
15 VTP.BASE 


343 25. | 1182 |__| =7.6589318579E+04 
199 914 ~~ |B | «1.2983146246E+05 


N 
Nn 


N 
oS 
wW 


16 | SC205 409 203. «| 552. | ~—s | —5.2202061212E+01 
BEACONFD | 436 262 | 3476 | 3.3592485807E+04 
18 | GROW7 2633 'B | —4.7787811815E+07 


19 LOTFI 
20 BRANDY 


4 08 |108 | — |~2,5264706062E+01 
470 249° 2150 =| ~—s | «1.5185098965E+03 


— 


a) 
w > 
2/9/8/8 
iw) 
x 
w 
os 


ala} a}alaja £8 & 
CO;CoO;N|/ BR) Wl Nn lon oO 
welSlolal|alo ean) iS 

a 

& 

— 

Ww 

=) 

= 


22 

21 | E226 506 = |224. «1282-12767: ~Sfsi«Y|sSSs| -1.1638929066E+01 
22. | BORE3D 549 | 23413151 1525, s«|| BSS s«1.3730803942E+03 
23. |FORPLAN |583 | 162,—/|421_:—«| 4916. =| BR_| —6.6421896127E+02 
24 | CAPRI 625 (272 |353 11786 |B | 2.6900129138E+03 
25 489 =|163  |2541 =| ~——| —3.5991767287E+07 
26 | BOEING1 352 «|384-— | 3865 S| BR._| —3.3521356751E+02 
27. | SCORPION 389/358 |1708 | — | 1.8781248227E+03 
28 306 «|472 |2659 =| Ss | -1.5862801845E+02 
29 | SCTAPI 301 «|480 «| 2052 =| ~—s | «1.4122500000E+03 
30 | SCFXM1 331 [457 2612 = | ~~ | —s«1.8416759028B+04 
31 | AGG3 819 |517 |302 (4531 =| ~~ | :1.0312115935E+07 

51 2 


Z 
819 30 


32 AGG2 


4515 | | =2.0239252356E+07 


(continued) 
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Table C.1 (continued) 


No. [Name [mtn [m |n _[Nonzeros [BR | Optimal value 

33 [STAIR | 824 [357 | 467 | 3857 _[B __| -2.5126695119E+00 
a4 [scsp1__| 838 | 78 | 760 | 3148 | | 8.6666666743E-.00 
35 [TUFF | 921__|334_| 587_| 4523 _(B_| _2.9214775747E-01 
36 [GROWIS | 946 [301_| 645 | 5065 |B | ~1.0687094129E+08 
37 _[SCAGRIS_| 972 [472 | 500 | 2029 | | —1.4753433061E+07 
38 [DEGEN2 | 979 _[44s_| sa4_| 4449 |_| =1.43517800006+03 
39 [FITID __|i0s1__| 25/1026 [14430 |B | -9.1463780924E.03 
40 [ETAMACRO_ [1089 _|401__| 688_| 2489 |B __|—7.5571521687E+02 
a1 [FINNS [1112 [498 | 614 | 2714 |B | _1.7279096547E105 
42 [FFFFFB00 [1379 _|525_| 854 | 6235 |__| _5.5567967533E105 
43 [GROW22 [1387 [441_| 946 | 8318 |B | ~1.6083433048E+08 
4 [PILOTS _[14i1_|411__|1000_| 5145 |B | ~2.5809984373E+03 


45 |STANDATA 1/1435  |360 | 1075 | 3038 = =|B | 1.2576995000E+03 
46 | SCSD6 1498 (148 |1350 | 5666 | — | 5.0500000078E+01 
47 |STANDMPS | 1543 | 468 | 1075 | 3686 |B | 1.4060175000E+03 
48 | SEBA 1544 |516 | 1028 | 4874 ~=— | BR_|| _‘1.5711600000E+04 
49 | STANDGUB 362, | 1184 | 3147 = |B |«1.2576995000E+03 
50 | SCFXM2 1575 «661 =| 914 | 5229 =| ~— | 3,6660261565E+04 
51 | SCRS8 1660 (491 |1169 | 4029 | ~— | 9.0430601463E+02 
52. |GFRD-PNC |1709 |617 | 1092 | 3467 |B | 6.9022359995E+06 


53 | BNLI 1819 644 «| 1175 | 6129 | S| s«1.9776292440E+03 
54 | SHIPO4S 1861 |403 | 1458 | 5810 | ~~ | ‘1.7987147004E+06 
55 | PEROLD 2002 | 626 =| 1376 =| 6026 =| B_| —9.3807477973E+03 
56 |MAROS 2290 (847 | 1443 |10006 |B |—5.8063743701E+04 
57 | FITIP (2305-628 =| 1677 | 10894 =| B_s|«<9.1463780924E+03 
58 |MODSZKI  |2308 |688 | 1620 | 4158 |B | 3.2061972906E+02 
59 | SHELL (2312. -|537. «11775 «| 4900 = |B =«i1..2088253460E+09 
60 | SCFXM3 991 (1371 | 7846 | ~~ | 5.4901254550E+04 


61 | 25FV47 2393/8222) | 1571 | 11127. | ~—s|s«5.5018458883E+03 
62 | SHIPO4L 403 |2118 | 8450 | — | 1.7933245380E+06 
63 | QAP8 913. | 1632 | 8304 ~=6| ~——|_—-2.0350000002E+02 
64. |WOODIP —_|2839 | 2452594 | 70216 =| =~ |_—«1.4429024116E +00 


(continued) 
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Table C.1 (continued) 


No. [Name [mtn [m [n [Nonzeros [BR [ Optimal vale 

65 [PILOTIA | 2929 | 941 | 1988 | 43220 |B _ | ~6.1130535360E+03 
66 [SCTAP2 | 2971 | 1091 | 1880 | si24 | | _1.72480714295403 
67 [GANGES | 2991 | 1310 | 1681_| 7021 [B | =1.0958577038E+05 
6 [Scsb8_| 3148_| 398 | 2750.| 11334 |__| 9.0499999993E+02 
69 [PILOTNOV | 3148| 976 | 2172 | 13129 |B | ~4.4972761882E+03 
70 [SHposs | 3166 | 779 | 2387 | 9501 |__| 1.9200982105E+06 
71 [SIERRA | 326 | 1228 | 2036/9252 |B | _1.5394390927E+07 
72 |DEGEN3 | 3322_| 1504 | 1818_| 26230 |__| ~9.8729400000+02 
73 [PILOLWE | 3512_|723_| 2789 | 9218 |B | ~2.7201037711E+06 
74 [NESM | 3586| 663| 2923 | 13988 [BR | _1.4076065460E+07 
75 [SHIPIIS | 3915_| 1152_| 2763 | 10941 |__| _1.4892361344+06 
76 [SCTAP3 | 3961_| 1481 | 2480 | 10734 |__| 1.4240000000E4+03 
77 |STOCFOR? | 4189 | 2158 | 2031 | 9492 |_| ~3:9024408538E+04 
78 [CZPROB | 4453_| 930 | 3523. | 14173 |B | 2.18519669895+06 
79 [CYCLE | 4761__| 1904 | 2857 | 21322 [B | ~5.2263930249E+00 
g0|SHIPOSL | 5062) 79 | 4283| 17085 |_| _1,90905521146-+06 
si [PILOT | S09 | 1442 | 3652 | 14706 |B | ~5.5728790853E+02 
s2[BNL2 | Seid_| 2325/3489 | 16124 | | _1.81124044506+03 
83 |SHIPIXL | 6579 | 1152_| sa27_| 21507 |_| _1.4701879193E406 
4 [D6CUBE? | 6600 | 416 | 184 | 43888 | _|_3.1473177974E400 
85 [D6CUBE | 6600 | 416 | 6184 | 43888 |B | _3.1549166667E+00 
s6[PILOTST | 6914 | 2031 | 4883 | 73804 |B | _3.0171261980E402 
s7[D2Q06C | 7339 | 2172| S167 | 35674 |_| _1.2978421653E405 
88 [OREENBEA | 7798 | 2393 | saos_| 31499 |B | ~7.24613936306+07 
39 |WOODW | 9504 | 1099 | 840s | 37478 |_| _1.30447633316400 
90 [FIT2D___|10526 | 26 | 10500138018 |B | ~6.8464293007E+04 
1 [QAPI2 | 12049 | 3193 | 8856| aaa |__| 5.22894350S0E+02 
92 [S0BAUSB | 12062 | 2263/9799 | 29063__|B | _9.8724313086E-+05 
93 [MAROS-R7_|12545__| 3137 | 9408 [151120 |__| _14971851665E+06 
94 [FIT __|16526 | 3001 [13525 | 60784 |B | 68464293052E+04 
95 [DFLOOI | 18302_| 6072 _|12030 | 41873 |B | _1.1266503030E4+07 
96 [STOCFOR3 [32371 |16676 [15695 | 74004 |__| ~3:9976256537E+04 
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Table C.2 Kennington problems 


No. [Name [mtn [m [nw | Nonzeros [BR [ Optimal value 
KENT | 6009 | 3427 | 3602 | 11981 |B | ~6.7952044338E08 
cREC | 6747 | 3069 | 3678 | 16922 | | 2.S975116141E+07 
CREA | 1584 | 3517_| 4067 | 19054 |__| _2.35954105895+07 
‘PDS.02_| 10489 | 2954 | 7535 | 21252 [B | 2.8857862010E+10 
“Osa07 | 25068 | 1119 | 33949 | 167643 |__| 5.3572951730E+03 
‘KEN-IT | 36084 | 14695 | 21349 | 70354_[B | =6.9723820625E409 
PDS-06 | 38537 | _9882_| 28655 | 82269 |B | 2.7761037600E+10 
‘osa-ld_| 54798 | 2338 | 52460 | 367220 |__| _1.1064628448E+06 
‘PDS-10 | 65322__| 16559 | 48763 | 140063 [B | 2.67270949T6Es10 
‘KEN-13_| 71292 | 28633 | 42659. | 139834 |B _|=1.0257394789E+10 
“CRED | 78907 | $927 | 69980 | 312626 |__| _2.4454969898E+07 
12 [CREB | $2096 | 9649 | 72447 | 308542 | | 2.3129640065E407 
13. [OSa-30 [104375 | 4351 [100024 | 700160 |__| _2.1421398737E+06 
14 [PDS-20_ [139603 | 33875 [105728 | 304153 |B | _2.3821658610E+10 
15 [OSA-60 [243247 | 10281 [232966 1630758 |__| 4.00725060E+06 
16 [KEN-18 [259827 [105128 [154699 | 512719 |B | ~5.2217025287E-10 
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Table C.3. BPMPD problems 


No. [Name [mtn _[m _[n _[Nonzeros [BR | Optimal value 
RATIA | 12545 | 3137 | 9408 | 275180 | | 2.0743714157E+06 
NSCTI__|_37883_| 22002 | 14981 | 667499 |_| —3.8922436000E+07 
NSCT2 37985 | 23004 | 14981 | 686396 | — | —3.7175082000E+07 
ROUTING 44818 | 20895 | 23923 | 210025 |B | 5.9416502767E+03 
DBIRI 46160 | 18805 | 27355 | 1067815 | — | —8.1067070000E+06 
DBIR2 46262 | 18907 | 27355 | 1148847 | — | —6.1169165000E+06 
T0331-4L 477897 |B | 2.9730033352E+04 
NEMSEMM2 | 49077 | 6944 | 42133 | 212793 |B | 6.2161463095E+05 
SOUTHERN 54160 148318 |B | 1.8189401971E+09 
RADIO.PRIM 265975 |B | 1.0000000000E+00 
11 | WORLD.MOD2 220116 |B | 4.3648258595E+07 
6 B 
7 
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12 | WORLD 220748 6.9133457165E+07 
13. |RADIO.DUAL | 74971 | 8053 | 66918 | 328891 |B | —1.0000000000E+00 
14. | NEMSEMM1 1120871 |B | 5.1442135978E+05 
15 |NW14 123483 | 74 + | 123409 | 1028319 |B | 6.1844000000E+04 
16 | LPLI 164952 | 39952 | 125000 | 462127 |B | 6.7197548415E+10 
17 | DBICI 226436 | 43201 | 183235 | 1217046 |B | —9.7689730000E+06 
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Table C.4 Netlib problems without bounds and ranges 


No. Optimal value 

i [AFRO | 60 | 28 | 32 | 88 | —4.6475314286E+02 
2 |scsoB_ | 99 | S| 48 | 19 __| =7.00000000008+01 
3. |scsoa | 99 | 5st | 48 | 131 —6.4575077059E+01 
4 465 2.2549496316E+05 
5 521 —3.0812149846E+01 
6 730 —4,1573224074E+02 
7 281 —5.2202061212E+01 
8 474 —4.1131976219E+04 
9 553 —2.3313897524E+06 
10 =8.9664482186E+05 
ul —7.6589318579E+04 


12 552 —5.2202061212E+01 
13 3476 3.3592485807E+04 
14 1086 —2.5264706062E+01 
15 1.5185098965E+03 
16 = 1.1638929066E+01 
17 2541 —3.5991767287E+07 
18 1,8781248227E+03 
19 = 1,5862801845E+02 
20 1.4122500000E-+03 
21 1,8416759028E+04 
22 1.0312115935E+07 
23 —2.0239252356E+07 
24 3148 8.6666666743E+00 
25 = 1.4753433061E+07 
26 —1.4351780000E+03 
27 6235 5.5567967533E+05 
28 5.0500000078E+01 
29 5229 3.666026 1565E+04 
30 4029 9.0430601463E+02 
31 1.9776292440E+03 
32 1.7987 14TO04E+06 
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Table C.4 (continued) 


No. Optimal value 
33 5.4901254550E+04 
34 | 25FV47 11127 5.5018458883E+03 
35 | SHIPO4L 403 1.7933245380E+06 
36 | QAP8 13 2.0350000002E+02 
2 
2 


37 5 2594 70216 1.4429024116E+00 
38 | SCTAP2 1091 1880 8124 1.7248071429E+03 


9 
24 
39 | SCSD8 398 2750 11334 9.0499999993E+02 
7 
7 


393 
521 
3 
40 SHIPO8S 3166 7719 2387 9501 1.9200982105E+06 
3 
42 SHIP12S 3915 1152 2763 10941 1.4892361344E+06 
3 
062 
814 


41 1504 1818 26230 —9.8729400000E+02 
43 | SCTAP3 1481 2480 10734 1.4240000000E+03 
44 |STOCFOR2 | 4189 2158 2031 9492 —3.9024408538E+04 
45 | SHIPOSL | 5062 «| «(779 4283 17085 1.90905521 14E+06 
46 | 5814 | 2325 3489 16124 1.8112404450E+03 


48 | DoCUBE2 16 6184 43888 3.1473177974E+02 


49 2172 1.2278421653E+05 
50 1099 1.3044763331E+00 
51 | QAPI2 3193 44244 5.2289435056E+02 
52. | MAROS-R7 1.4971851665E+06 
53 74004 —3.9976256537E+04 


5 
5 
47 SHIP12L 6579 1152 5427 21597 1.4701879193E+06 
6 
7 
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Table C.5 Kennington problems without bounds and ranges 


Optimal value 

16922 | 2.5275116141E+07 
19054 | 2.3595410589E+07 
167643 | 5.3572251730E+05 
367220 | 1.1064628448E+06 
312626 —_| 2.4454969898E+07 
328542 | 2.3129640065E+07 
700160 | 2.1421398737E+06 
1630758 | 4.0440725060E+06 


No. Optimal value 

! 2.0743714157E+06 
2 667499 —3.8922436000E+07 
3 686396 | —3.7175082000E+07 
4 1067815 —8.1067070000E +06 
5 1148847 | —6.1169165000E+06 


Appendix D 
Empirical Evaluation for Nested Pivot 
Rules 


This appendix offers detailed numerical results, obtained from our extensive 
computational experiments on selected pivot rules, associated with those presented 
in Chap. 5 of Part I. 

Compiled using Visual Fortran 5.0, all codes were run under a Windows XP 
system Home Edition Version 2002 on an IBM PC with an Intel(R) Pentium(R) 
processor 1.00GB of 1.86GHz memory, and about 16 digits of precision. 

The test sets included three types of problems from Netlib, Kennington, and 
BPMPD test sets (see below). These problems are ordered by their sizes, in terms 
of m +n, where m and n are the numbers of rows and columns of the constraint 
matrix, excluding slack variables. 

In results listed below, all reported CPU times were measured in seconds with 
utility routine CPU_TIME, excluding the time spent on preprocessing and scaling. 
Final objective values reached are not listed. All runs were terminated within the 
primal and dual feasibility tolerance 10~°. 

MINOS was used as the testing benchmark and platform. To compare easy and 
fairly, each rule is implemented within MINOS 5.51, the latest version of MINOS 
at that time. Actually, all resulting codes are the same as MINOS 5.51, except for its 
pivot rule replaced by relevant ones. All codes used the default options, except for 

Rows 200000; 

Columns 300000; 

Elements 5000000; 

Iterations 20000000; 

Scale yes; 

Solution no; 

Log frequency 0; 

Print level 0. 


© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 703 
P.-Q. PAN, Linear Programming Computation, 
https://doi.org/10.1007/978-98 1-19-0147-8 
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(1) We first report results associated with the following five codes and make 
comparisons between one another: 


1. Dantzig: MINOS 5.51 with the full pricing option. 

2. Devex: Devex rule. 

3. P-Dantzig: MINOS 5.51 with the default partial pricing option. 
4. N-Devex: Nested-Devex Rule. 

5. N-Dantzig: Rule 12.5.1. 


A. Results for 48 Netlib problems 

The first set of test problems included the 48 largest Netlib problems. Associated 
results are listed in Table D.1, where total iterations and time required for solving 
each problem are listed in columns labeled Iters and Time for the five codes. The 
totals listed in the second bottom line are for the 47 problems, excluding QAP15, as 
it is too time consuming to solve QAP15 with Dantzig and P-Dantzig, and results 
with the other three codes would heavily dominate the entire computations. Results 
associated with QAP15 are listed separately in the bottom row of the table. 

To serve as a comparison between the codes, Table D.2 lists iterations and run 
time ratios of the first four codes to N-Dantzig. From the second bottom line, it is 
seen that N-Dantzig and N-Devex outperformed all the standard codes, in terms of 
both iterations and run time. N-Devex performed slightly better than N-Dantzig with 
time ratio 0.95. On the other hand, code Dantzig performed worst on this set with 
largest iterations ratio 5.16 and time ratio 6.00. It is also noted that Devex performed 
better than P-Dantzig. 

B. Results for 16 Kennington problems 

The second set of problems includes all 16 problems in Kennington test set. 
These problems are larger than problems in sets 1, overall. Associated numerical 
results are listed in Table D.3 and compared in Table D.4. From the bottom line 
of the latter, it is seen that N-Dantzig and N-Devex again defeated all the standard 
codes in terms of both iterations and run time. Again, N-Devex performed a little 
better than N-Dantzig with time ratio 0.91. Code Dantzig performed worst on this 
set with largest iterations ratio 5.63 and time ratio 5.65. In contrast to the first set, 
however, Devex performed worse than P-Dantzig with this set. 

C. Results for 17 BPMPD problems 

The third set consists of the 17 largest BPMPD problems, in terms of more than 
500KB in compressed form. Associated numerical results are listed in Table D.5 and 
compared in Table D.6. From the bottom line of Table D.6, it is observed that the 
codes performed similarly to the first two sets. Even with larger margins, N-Dantzig 
and N-Devex defeated the standard codes in terms of both iterations and run time. 
In fact, N-Dantzig performed best with this set while code Dantzig did worst with 
iterations ratio 8.29 and time ratio 12.86! 
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Table D.2 Ratio to N-Dantzig for 48 Netlib problems 


Problem 
SCRS8 
GFRD-PNC 
BNLI 
SHIPO4S 
PEROLD 
MAROS 
FITIP 
MODSZK1 
SHELL 
SCFXM3 
25FV47 
SHIPO4L 
QAP8 
WOOD I1P 
PILOT.JA 
SCTAP2 
GANGES 
PILOTNOV 
SCSD8 
SHIPO8S 
SIERRA 
DEGEN3 
PILOT.WE 
NESM 
SHIP12S 
SCTAP3 
STOCFOR2 
CZPROB 
CYCLE 
SHIPO8L 
PILOT 
BNL2 
SHIP12L 
D6CUBE 
D6CUBE2 
PILOT87 
D2Q06C 
GREENBEA 
WOODW 
TRUSS 


PDanwzig | N-Deves 

ters [Time 
091/050 
1.06 | 1.00 
1.10 | 1.00 
(0.68 [1.00 [0.66 [1.00 [074 [1.00 [1.02 | 1.00 
0.98 | 1.00 
051 [062 [04s [054 [074 [0.69 0.77 [0.09 
‘094 [1.00 [0.59 [060 [089 [080 [097 [0.80 
078 [133 [0.66 [067 [1.05 [1.00 [097 | 1.00 
1.00 | 1.00 
(066 [083 [0.66 [083 [078 [083 [0.96 [083 
0388 |0.89 
10.66 [1.00 [0.67 [1.00 [0.76 [1.00 [1.01 | 1.00 
1.01 [1080.53 _[057 [099/097 [085 [08 
078 |0.86 
1.06 [1.12 
1090 [1.00 [1.00 [1.33 [1.05 [1.00 [1.04 [1.00 
1.06 | 1.00 
1.o1|107 
098/083 
097 [200 [091 [200 (099/100 [1.02 | 1.00 
1.01 [1.00 
087 [0.88 
1.07 | 1.09 
124 [124 
091 [150 [09 [1.50 (099/100 | 1.01 1.00 
083 [100 [092 [120/101 |1.00 [099 [0.80 
041 [048 [0.38 [044 [053 [056 [0.90 [0.88 
1.06 [1.20 
0.94 |0.96 
1.00 | 1.00 
060 [078 [040 [055 (076 [081 (0.98 (0.89 
1.02 [138 [071 [096 [087 [091 [0.95 [093 
1.04 | 1.00 
128 [1.24 
089/083 
1.03089 
122 [1.17 
107/103 
1.04 [107 
1.02 0.98 


(continued) 
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Table D.2 (continued) 


Problem ites 
FIT2D 0.97 
QAP12 
S0BAUB 1.02 
MAROS-R7 119 
FTP (097 (121 [0650.69 [1.14 [118 | 1.02 
DFLOOI 1d 
STOCFORS [0.77 [1.02 [078099 [0.91 0.98 | 1.00 
Average [5.16 [600 [120 [1.21 [4.65 [4.00 [1.04 
qapis|- _|- _|09s_ loa? |- |- 088 
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Table D.4 Ratio to N-Dantzig for 16 Kennington problems 


Problem ters 
KENOT | 1.00 | 131 | 096 | 1.15 [1.06 [108 | 1.00 
CRE-C 1.03 
CREA | 096 | 120 | 071 | 090 [086 [086 | 1.00 


PDS-02 
OSA-07 


KEN | 100 | 134 | 096 | 195 [14s [118 [1.01 
PDS.06 1.00 
OSA-14 1.05 
PDS.10 125 
KEN-13 1.00 


CRE-D 
CRE-B 
OSA-30 
PDS-20 


OSA-60 1.02 
KEN-18 1.00 
Average 1.00 
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rs9[ 0606) Sr] _soas| _rer| 1st] "zz | __cor| 
osc] ruo0c| eer] riser] sei] owe) _e6) ores) 


specie | waserse] secre] OnmraL 
erusest | oseeses | scerrer | (core 
erase | wereo| seer [tora 
oeoces | zrieree | zseron [a 
re tor exec [pr 
toe | oseer | esest | TNS 
60f | tuere | trorave 
Eistor|saeteee | —srew [arrow 
essaez | —isisorr| geet | aWarrIOm 
rtf a) st099 | arora 
seer | se6e|—optrs | NwaHLLNOS 
rest | 1asez | —u206r | anvasnian 
—o'se9 | geez | oxser [recon 
wets | sistr| oor [aaa 
ser eaze| opin [rata 
~Fteor | iaa06t [srr | ONL 
eoor|ssstr|seeue [aD SN 
(ete|__oste|exace [11s 
ess | eto | srscr | vaava 
au [Sena woqgou 


swiajqoid GdqINd LI J0J sonsneig s*C age, 
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Table D.6 Ratio to N-Dantzig for 17 BPMPD problems 


PDanwzig | N-Devex 
Problem ters | Time 
RATTA 0.94 [0.63 
NSCT 0.65 [0.70 
NSCT2 1.06 | 1.03 
ROUTING 084 [0.85 
DBIRI 097 | 106 
DBIR2 | 531_ [1357 [163 | 412 [672 [7.77 [096 [0s 
0331-41 091 [0.98 
NEMSEMM2 14 [1.45 
SOUTHERN 1.01/09 
RADIO.PR 1.00 | 1.00 
WORLDMD | 6.38 | 7.75 (194 | 241 [563 [587 (099 [097 
WORLD 094 [0.92 
RADIO.DL 1.00 | 1.00 
NEMSEMMI 134 [1.35 
NWI4 1.06 [1.13 
LPLI 1.70 [1.64 
DBICI 131 [145 
Average 1s [1.22 


D. A summary 

Table D.7 offers ratios of the first four codes to N-Dantzig for each test set and 
for all (excluding QAP15). It is seen that the codes performed consistently with the 
three sets, overall. Further, N-Dantzig and N-Devex defeated the standard codes in 
terms of both total iterations and run time. N-Dantzig even outperformed Devex rule 
with total iterations ratio 3.48 and average time ratio 5.73. It can be asserted that N- 
Dantzig performed best overall while code Dantzig did worst with iterations ratio 
6.78 and time ratio 9.75. 

The ratio of the number of iterations required by a code to the number of rows and 
columns should be an indicator for problem’s difficulty to the code. The smaller the 
ratio is, the easier the problem solved by the code. Following Forrest and Goldfarb 
(1992), we, therefore, list in Tables D.8 and D.9 normalized iterations required by 
the five codes for the Kennington and BPMPD test sets, respectively. 


Table D.7 A ratio summary 


P-Danvig 
Problem hers | Time 
Neti) 1.04 (0.95 
Kemington(16) 1.00 [0.9 
BPMPD(17) 118 [1.28 
Average(80) 1.10109 
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Table D.8 Normalized iterations for 16 Kennington problems 


No N-Dantzig 


1 [KENO7 | 029 | 027 [0301029 ——~*4(028 
2 cee [00 | 04s Joes fos 0.68 
3 [CREA | 054 | 039 ~[oas——*(ios6 ——*(0.56 
4 0.19 [0.20 
5 |osao7 | 007 | 007 [00s [006 0.06 
6 037 [a7 
1 os _[0.34 
s[osai4 | 007 | 007 [008 —~(006 0.06 
9 046 [037 
io |KEN-3| 050 | 041 [066 [046 [0.46 
1 |cRED | 478 —*(| 527 —*(35——~iog0——=*d 
2 077 [078 
13 _[osaso | 007 | 007 [o0s_—~(006 0.06 
14 ost [067 
i5_[osa6o | 007 | 007 [007 —~(006 [0.06 
16 053 (033 


040 [0.40 


No N-Dantzig 
! 1.50 1.60 
2 |NscT! | 0.06 | 0.04 | 0.06 0.15 0.24 
3 0.16 0.16 
4 0.42 0.50 
5 0.26 0.26 
6 0.17 0.17 
7 0.95 1.05 
8 0.28 0.21 
9 0.44 0.44 


10 |RADIO.PR | 0.00 __| 0.00 | 0.00 __—|0.00___—| 0.00 
u 3.23 3.27 
12 3.83 4.08 
13 |RADIODL | 0.00 | 0.00 | 0.00 0.00___—| 0.00 
4 0.17 0.13 


is [NWis | 0.00 | 000 | 001 [oor (00 
16 270 (159 
045 [035 
rAverage «| 674 =| a * at —~«idio6——=do 
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From the bottom lines of the Tables, it is seen that, on average, normalized 
iterations required by N-Dantzig and N-Devex are less than one, whereas those 
required by their standard counterparts are greater that two. In particular, normalized 
iterations required by N-Dantzig and N-Devex are less than one for each Kennington 
problem. Overall, normalized iterations required by them are quite stable, while 
those required by their standard counterparts fluctuate. 

There is no doubt that nested pricing rules are amenable to the test sets, as is the 
case can be expected in general. 

(2) We also report our computational experiments with the following three codes: 


1. Steepest-Edge(Stp): the Steepest-edge rule. 
2. N-Steepest(N-Stp): Nested-steepest-edge rule. 
3. N-Dantzig(N-D): Rule 12.5.1. 


Computational results associated with Netlib, Kennington, and BPMPD prob- 
lems are given in Tables D.10, D.11, D.12, D.13, D.14, D.15. The first two codes 
failed to solve QAP15 in Netlib, KEN-18 in Kennington, and LPL1 and DBIC1 in 
BPMPD within 48 hours. So, we report results, excluding the four problems. 


Table D.10 Statistics for 47 Netlib problems 
Stp N-Stp N-D 

Problem ites | Time 
SCRS8 0.2 
GFRD-PNC 0.2 
BNLI od | 08 | 98 || 1509 | 
SHIPO4S 0.1 
PEROLD 18 
MAROS 1.3 
FITIP | 39 | 06 | 804 | || 
MODSZK1 0.3 
SHELL 232 244 275 0.1 
SCFXM3 0.6 
25FV4T ee ee ee 2.7 
SHIPOAL 0.1 
GAPS —3sis [99 | asi) 39 ie 10.7 
WOODIP 0.7 
PILOT | igor | 38 | tia | 80 | 56733 
SCTAP2 733 0.3 
GANGES | 571 | 06 | 727 | 8 | 99 | 
PILOTN 14 
SCSD8 0.6 


(continued) 
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Table D.10 (continued) 


Problem 
SHIPO8S 
SIERRA 
DEGEN3 
PILOT.W 
NESM 
SHIP12S 
SCTAP3 
STOCF.2 
CZPROB 
CYCLE 
SHIPO8L 
PILOT 
BNL2 
SHIP12L 
D6CUBE 
D6CUBE2 
PILOT87 
D2Q06C 
GREENBE 
WOODW 
TRUSS 
FIT2D 
QAP12 
80BAU3B 
MAROSR7 
FIT2P 
DFLO01 
STOCF.3 
Total(47) 


Stp N-Stp N-D 
ters 


7 | 06 | 27 | 06 | 28 


im [22 | 960 | 27 | 907 
900 [2s | io | 28 | 1420 
170669 


600071 


715 


Time 
0.1 
0.5 
3.3 
2.2 
2.1 
0.2 
0.5 
2.5 
0.5 
2.4 
0.3 

38.3 
4.5 
0.5 
5.8 

11.1 

97.2 

20.5 

13.2 
2.9 
8.5, 
8.5 

1454.1 

9.0 
36.2 
37.0 

398.6 

78.8 

2264.7 
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Table D.11 Ratio for 47 Netlib problems 


Problem 
SCRS8 
GFRDP 
BNL1 
SH04S 
PEROL 
MAROS 
FITIP 
MODS1 
SHELL 
SCFX3 
25F47 
SH04L 
QAP8 
WOOD1 
PIL.J 
SCTA2 
GANGE 
PLTNO 
SCSD8 
SHO08S 
SIERR 
DEGE3 
PIL.W 
NESM 
SH12S 
SCTA3 
STCF2 
CZPRO 
CYCLE 
SHO8L 
PILOT 
BNL2 
SH12L 
D6CUB 
D6CU2 
PIL87 
D2Q06 
GREEN 


N-SipIN-D 
lies | Time 
0.84 | 2.56 
Lol | 2.24 
2.32 
403 [i458 | 0.99 | 100 | 064 | 213 loo | 2.13 
626 0.58 | 1.86 
847 | 1443 | 0.65 | 0.66 | 0.35 | 1.39 0.54 | 2.10 
628 0.85 | 2.28 
688 0.76 | 3.70 
537 0.89 | 2.75 
991 0.67 | 2.24 
822 2.41 
403 3.00 
913 0.39 | 1.30 
245 1.10 0.74 | 4.00 
941 0.47 2.42 
1091 0.78 5.48 
1310 0.81 1.04 | 2.52 
976 0.51 0.82 | 3.56 
398 0.44 0.70 | 4.32 
7719 1.04 1.03 | 4.31 
1228 {2036 | 099 | 1.01 3.88 
1504 O71 | 3.03 
723 0.41 0.63 | 3.77 
663 0.38 0.80 | 4.90 
1152 0.88 1.07 | 5.59 
1481 0.82 1.06 | 5.79 
2158 0.73 2.12 
930 084 | 5.28 
1904 0.39 5.06 
7719 1.00 [069 | 7.24 |o91 | 7.24 
1442 0.79 0.71 | 8.35 
2325 0.67 0.58 | 3.77 
1152 {5427 | 089 | 096 | 093 | 882/104 | 9.18 
416 0.68 | 7.76 
416 5.64 
2031 7.81 
2172 0.58 | 5.73 
2393 7.48 


(continued) 
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Table D.11 (continued) 


N-SipiN-D 
Problem ners [Time 
wooDW 073/132 
TRUSS 089/136 
FIT2D 050 | 674 


QAPI2 0.42 | 5.22 
80BAU___| 2263/9799 | 0.57 | 0.51 | 0.34 | 5.36 |0.60_| 10.59 
MARR7 100 | 11.31 


FIT2P 086 [11.29 
DFLO0 5.40 
ST003 1.01 (20.45 
ave [= |- | 060 | ost [oa | 3u8 6.19 


Table D.12 Statistics for 15 Kennington problems 


Problem lies | Time 
KEN-07 to | 6.0 | 1723, | | TB 13 
CRE-C 1391 4345 44 
CRE-A 1497 4228 4.9 
PDS-02 911 2099 1.8 
OSA-07 1507 1504 1.5 
KEN-11 11931 13361 61.7 
PDS-06 3304 13100 40.6 
OSA-14 3180 3212 6.0 
PDS-10 6562 24179 134.1 
KEN-13 23769 33074 339.5 
CRE-D 17782 73528 312.1 
CRE-B 16411 63703 277.1 
OSA-30 6254 5963 20.5 


PDS-20 24854 52549.0 28443 59942.4 93045 1199.4 
OSA-60 12809 24627.3 16731 30928.0 13751 112.1 
Total(15) 133951 109945.9 161805 132042.2 350810 2517.0 


Table D.16 offers ratios for each test set and for all the 77 problems as a whole. It 
is seen that N-Dantzig outperformed Steepest-Edge with average time ratio as high 
as 25.22 despite it required more total iterations (with iterations ratio 0.34). It is also 
noted that Nested-Steepest-Edge was inferior to Steepest-Edge. 

In summary, the nested pricing are unambiguously superior to standard one. As 
the best among nested ones, the nested Dantzig criteria should be used whenever 
possible. 
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Table D.13_ Ratio for 15 Kennington problems 


N-SipiN-D 
Problem ners [Time 
CREC 5.10 
CREA 5.91 
PDS-02 7.02 
osao7 [119 [23949 [096 [1.03 [1.00 | 27.11 [104 | 36.34 
KEN-I 092 | 17.69 
PDS-06 oat [1437 
Osa-lé [2338 [52460 [099 [099 [099 | 5569 [1.00 | 56.13 
pDS-10 035 | 2565 
KEN-13 077 | 290 
CRED 030 | 32.89 
CREB 038 | 46.07 
OSA-30 130 [13302 
PDS-20 031 | 49.58 
OSA-60 122 [275.82 
Avenge [- |= sos Jose | 43.68 [os | 52.46 


Table D.14 Statistics for 15 BPMPD problems 


sp INS ND 
Problem iwers | Time 
RATTA 29638 
NSCTI 65.1 
NSCT2 228 
ROUTING 124.5 
DBIRI 52.9 


DBIR2 60.4 
T0331-4L 282.1 
NEMSEMM2 30.0 
SOUTHERN 318.1 
RADIO.PR 14 
WORLD.MD 220329 3077.7 
WORLD 278284 _|_3899.0 
RADIO.DL 0.9 
NEMSEMM| 30.9 
NW14 5.2 
Total(15) 670592 8317.8 
Total(77) 1621473 | 13099.5 
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Table D.15_ Ratio for 15 BPMPD problems 


N-SipiN-D 
Problem ners [Time 
RATTA 5.05 
Nscri_|2902. [49st [0.99 oo (022 | 831 [022 | 8.80 


NSCT2 0.71 | 26.34 
ROUTING | 20895 | 23923 0.95 |0.96 [0.36 | 18.88 |0.37_| 19.76 
DBIR1 0.20 | 13.15 
DBIR2 0.26 _| 13.90 
T0331-4L 0.44 | 21.38 
NEMSEMM2 0.89 | 63.89 
SOUTHERN | 18739 [35421 0.76 |0.73 10.69 | 12.60 |0.91 | 17.26 
RADIO.PR 1.00 | 21.82 
WORLD.MD 0.50 _| 43.00 
WORLD 0.47 _ | 42.03 
RADIO.DL 1.33 | 38.75 
NEMSEMMI 0.76 _ | 73.60 
NW14 74 [123409 [o69 [090 [0.29 | 244 [043 | 2.71 
Average |= [- 067 0.67 [033 [2563 [0.50 [38.42 
Table D.16 A second ratio N-Stp/N-D 

Ree Problem Iters | Time 


Netlib(47) 0.60 |0.51 |0.33 | 3.18 |0.55 | 6.19 
Kennington(15) 0.46 | 52.46 
BPMPD(15) 0.50 | 38.42 
Average(77) 0.51 [35.55 


Appendix E 

Empirical Evaluation for Primal and 
Dual Face Methods with Cholesky 
Factorization 


To give an insight into the behavior of face methods, the following three codes in 
FORTRAN 77 were tested and compared: 


1. RSA: The revised simplex algorithm supported by the classical Phase-I (see 
Notation). 

2. FALP: Face Algorithm 21.6.1, supported by Algorithm 21.6.2, with components 
of the initial x being all ones. 

3. DFA: Dual face Algorithm 22.6.1, supported by Algorithm 22.6.2. 


All the preceding were dense codes in the sense that they did not exploit sparsity 
structure. Harris’ two-pass practical tactic was used. Number 10~° was taken as 
primal and dual feasibility tolerance, and 10~* as equality relative tolerance. In 
FALP, ||Aglloo < 1078 was used in place of Ag = 0, while in DFA, ||Aylloo < 
10-8 used in place of Ay = 0. 

Compiled using the Visual FORTRAN 5.0, all codes were run under a Windows 
XP system Home Edition Version 2002 on an IBM PC with an Intel(R) Pentium(R) 
processor 1.00GB of 1.86GHz memory, and about 16 digits of precision. All 
reported CPU times were measured in seconds with utility routine CPU_TIME. 

Tested was a set of 26 standard problems from NETLIB that do not have 
BOUNDS and RANGES sections in their MPS files. Specifically, these were the first 
26 such problems in the order of increasing sum of numbers of rows and columns 
of the constraint matrix, before adding slack variables (see Table C.4 in Appendix 
i). 

Test statistics obtained with RSA, FALP, and DFA are listed in Tables E.1, E.2, 
and E.3, respectively. Total iterations and time, required for solution of each 
problem, are displayed in columns labeled Itn and Time, respectively, and those 
associated with Phase-1 alone in columns labeled Itn! and Time1l. Percentages of 
total degenerate iterations are listed in columns labeled % Deg; those associated 
with Phase-1 are in the column labeled % Deg1. In the last column labeled k-m of 
Table E.2, listed are the dimensions of the optimal face finally reached, whereas the 
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Table E.1 Statistics for code RSA 


Total Phase 1 


Problem Tm [Time [%Deg__|ltnl___|Timel__| Deg 
AFIRO | 29 | 003 69.0 | 8 | 003d 7h 
SC50B | 57 | (0.06 80.7 | 50 | 0.06 | 90.0 
SC50A | 55 | 009 74s | St | 0.09 | 78.4 
ADLITTLE | 134 | 0.13127 | 69 | 01 24.6 
BLEND “as [017 [400 | 90) —* sit 
SHARE2B 196 | 028 [staf a9 —*| 027 fora 
SC105 | 123 | 036 73.2 | to | 036 177.3 
STOCFOR1 ptm | 050 feed 50 048 |79.3 
SCAGR7 | igi | 067 343 | 146 | 064d 42S 
ISRAEL [sm | iat} os [or 
SHAREIB Ee 
SC205 255 | 2.23 66.7 | at | 216 | 80.1 
BEACONFD ee ee ee 2.53 |57.9 
LOTFI | 354 | 313 toa | 190 | 2.91 | 22.6 
BRANDY Sse ase fst fa 3.63 |49.2 
£226 | ols | 550 123.7 | 394 | 494 | 32.7 
AGG po [iset | 6a [st 12.89 |_7.0 
SCORPION | 405 | 16.64 61.0 | 380 | 16.47 __—*d 64.5 
BANDM | 666 | 2042 218 | 467 | 19.39 | 30.0 
SCTAPI | 481 | 23.11 [36.6 =| 349 22.44 44.4 
SCFXMI ser 2698 [ast | 429 | 26.0355 
AGG2 ae ee ee 5.6 
AGG3 | 837 | 52.14 5.5 | 627 | 49.7 6.1 
SCSDI | 86 | 52.3187 | 80 | 52.23 | 97.5 
SCAGR25 ae ae = 60.42 44.5 
Total 25) 316.69 [47.3 
DEGEN2 1990 87.84 45.7 


column labeled m-k in Table E.3 are dimensions of dual optimal faces. Objective 
values attained are not listed—all runs were terminated within the feasibility 
tolerances except for problem DEGEN2, which RSA failed to solve within 10000 
iterations, and the associated results were listed separately, in the bottom lines of the 


tables. 


(1) A comparison between RSA and FALP. 

Table E.4 gives iteration and time ratios of RSA to FALP. From the second 
bottom line, it is seen that the total iteration ratio with the 25 test problems is 1.51, 
while the total time ratio is 4.91! As the time ratio with SCSD1 is exceptionally high 
(83.03), the bottom line lists total ratios with the 24 problems, excluding SCSD1. It 
is seen that the associated iterations and time ratio are 1.69 and 4.17, respectively, 
as indicates that results with SCSD1 alone do not dominate too much. 
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Table E.2 Statistics for FALP 


Problem feDegl | kom 
AFRO | 2% | 002 (00 | 7 | 000 [00 | 9 
scsop+| 30/002 [00 | 1 ~| 000 | 00 | 20 
sco +| -32~—~(| 002 -|00~~|| ~-1~—*«| 0.00 —*| 00 _—*| 20 
ADLITILE | 175 | 0.13 [00 | 4 | 003 | 00 | 15 
BLEND | 49 | 008 [00 | 11 | 003 [00 | 44 
SHARE2B | 85 | 006 [00 | 35 | 003 | 00 | 14 
SC105 6 [005 [oo | 1 [om [00 | 45 
STOCFORT | 66 | 009 [00 | 23 | 003 | 00 | 
SCAGRT | 14 | 023. [00 | 91 | 043 | 00 ‘| 84 
ISRAEL | 294 | 067/03. ~«| -9~(| 016 [00 | 0 
SHAREIB | 243 | 034 [00 | 140 | 016 | 00 | 89 
C205 P1393 [oss [oo | 1 [008 [00 | 91 
BEACONFD | 135 | 044/30 ~(| 60/016 ~(| 50144 
LoTR | «353/073 ~(06~—~*«|s8 ~~ 030 ~—«| 0S 
BRANDY | 246 | 091/12 ~*(|-7%6 ~~/| 023*| 00 (1a 
£226 vase [isi [oz [192 | 053 [00 | 7 
EE 160 [445 [oo | 9 [181 [oo | #9 
SCORPION 74 _[249 
BANDM | 410_| 5.02/00 | 167_| 155 | 00 (308 
scrap) «| 49 «355.29~~=«| | 02+ 00 120 
ScFxMI | 528 | 623 [02 +(383~(|:3.78 | 00191 
AGG? (280 [ 797 [oo | 67 | 214 [00 | 76 
AGG "290 [825 [oo | 86 | 250 | 00 | 76 
scspi__+| 736 ~| oes [98 | 1 | 003 [oo | 7 
SCAGRIS | 556 [23.06 (00 | 361_—~(1347 | 00 300 
Towal2s) (6088 (67.65 [09/2270 [2880 | 09 | 94 
DEGEN? | 452 [1113 [00 | 190 | 394 | 00 (219 


(2) A comparison between RSA and DFA. 

Table E.5 gives iterations and time ratios of RSA to DFA. If is seen from the 
second bottom line that the total iteration ratio with the 25 test problems is 1.19, 
while the total time ratio is as high as 10.04! As the time ratio with SCSD1 is as 
exceptionally high (1046.20), the bottom line lists total ratios with the other 24 
problems: the associated iteration and time ratios are 1.17 and 8.47, respectively. 

Considering that the time ratio is the only indication for efficiency, we conclude 
that the primal and dual face codes outperformed RSA unambiguously. 

Such outcome might be astonishing, but not surprising: 

On the one hand, FALP solves a single triangular (m + 1) x (m+ 1)-system in 
each iteration, compared with RSA solving four triangular m x m-systems. 
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Table E.3 Statistics for code DFA 


Problem mek 
AFIRO | 30 | 000/200 | 23 | 00 | 00 {3 
scsop =| -50~—~| 00 | 00 | #8 | 000 | 00 [2 
scsoA—*|-53~—*(| 00 | 00 | 47 | 000 | 00 [1 
ADLITTLE | 164 | 003 [152 | 29 | 002 | 00 [0 
Bueno [rotons | a9 asf om 00 
SHARE| 142_| 003 [113 | 69 | 002 | 14 

scios | 10 [aos oo | wo om [00 [1 
STOCFORI | 152 | 008 | 00 | 127 | 005 | 00 | 
SCAGRT | 254 | 023 [18 | 10 | 006/109 | 
ISRAEL | 321 036/128 | 140 | 011 | 00 [0 
SHAREIB | 245 | 013 | 45 | 119 | 006 | 00 | 
seus | 209 | 027 | 00 | 202 | 020 [00 [2 
BEACONFD | 136 | 006279) ~«1~~(| 000 | 00 [0 
tort | 264 | om? | 83_| a7 | 013 | 00 _|o 
BRANDY | 556 | 089 (3647 | 1 | 000 | 00 [16 


£226 381 | 07 | 73 | 156 | 0.23 | 00 | 
AGG 542 | 359 | 9.0 | 122 | 068 | 08 


SCORPION | 362 | 144 [22.1 | +1 ~(| 000 | 00 ‘(21 
BANDM 03 (0 
scrap! | 365 | 075 (790 | 1 | 00 | 00 {33 
SCFXMI 05 [I 
AGG? csi [497 [53 [161 [125 | 00 fo 
AGGS vse [si4_ [sa isr [iia | 00 _|o 
scspi_+| 78 (| 00s 359 | 1 ~~ 000 | 00 ‘0 
SCAGRIS | 822 | 789 (168 | 360 | 275 (13 

Total25) i 

DEGEN? [1627 [1809 | 60 | 44 | 3.53 | 02 


On the other hand, it is seen from the last column of Table E.2 that all optimal 
faces, finally reached by FALP, are not vertices, except for ISREAL, and that the 
average dimension of these faces is about 94. The ratio of average dimensions to 
average rows is as low as 94/213 = 0.44. In general, a high-dimensional optimal 
face is easier to achieve than an optimal vertex. 

As for a comparison between the two face codes, DFA is even superior to FALP, 
partially due to that the former solves a smaller triangular k x k system in each 
iteration, where k varies dynamically but never exceeds m. From the last column of 
Table E.3, in fact, it is seen that k did not reach m for 14 out of the 26 test problems. 
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Table E.4 Ratio of RSA to FALP 


Total Phase 1 
Problem m n m+n Itn Time 


AFIRO 28 | 32 | 60 2 | 150 | 4.00 | 300.00 
SC50B 51 | 48 | 99 1.90 | 3.00_| 50.00 _| 600.00 
51 


scoa | St«| a8 iT | aso | 51.00 | 90000 
ADLITTLE 433 
BLEND 5.61 
SHAREZB 9.00 
SCI05 ios [103 [209 [1.89 | 600 [110.00 | 1800 
STOCFORI 16.00 
SCAGRT | 130 402 
ISRAEL 6.06 
SHAREIB 9.88 
C205 206 27.00 
BEACONFD 15381 
LOT «isd 308 [462 ——(1.00—~| 429 ~*| 096 | 9:70 
BRANDY 15.78 
£226 9.30 
AGG 489 2 
SCORPION 77.00 
BANDM 1251 
SCTAPI 22.00 
SCFXMI 6.89 
AGG? 1715 
AGG3 19.61 
SCSDI 80.00. [1741.00 
SCAGRI5 4.49 
Average(25) 11.00 
Average(24) [218 9.19 
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Table E.5 Ratio of RSA to DFA 


Total Phase 1 
Problem m n m+n Itn Time 
28 


AFIRO 28 60 [0.97 | 127 - 

SC50B ee ee Ee | 104 |- 

SC50A a ee | 109 | - 

ADLITTLE 97 4.33 238 | 6.50 

BLEND 83 5.67 1.20 | 8.50 

SHARE2B 79 9.33 2.16 | 13.50 
2 


SC105 106 103 12.00 1.08 | 18.00 


STOCFOR1 111 229 6.25 1.18 | 9.60 
SCAGR7 130 | 140 2.91 1.33 | 10.67 
ISRAEL 142 3.92 1.34 | 8.82 
SHAREIB 225 12.85 1.60 | 26.33 


SC205 206 = | 203 8.26 1.04 | 10.80 


mw] ~ ISESES 
Ol] se a nr) oo) uw 
Q| oo S A/S} ns) 


BEACONFD 262 43.50 : 
LOTFI 308 11.59 22.38 
BRANDY 249 4.34 : 
E226 282 7.05 21.48 
AGG 489 163 3.80 20.46 
SCORPION 358 11.56 : 
BANDM 143 15.89 
SCTAPI 30.81 1122.00 
SCFXMI1 8.18 37.19 
AGG2 302 7.94 29.36 
AGG3 302 10.14 43.13 
SCSDI 1046.20 : 
SCAGR25 500 8.18 21.97 
Average(25) 10.04 36.70 
Average(24) 218 8.47 30.64 


Appendix F 
Empirical Evaluation for Simplex 
Interior-Point Algorithm 


Computational experiments have been performed to gain an insight into the behavior 
of Algorithm 25.3.1. 
Implemented and compared are the following three codes: 


1. MINOS: MINOS 5.51 with full pricing. 
2. FPS: Two-Phase code based on Algorithm 25.3.1. 
3. FPSP: Two-Phase code based on Algorithm 25.3.1 with the purification. 


In both FPS and FPSP, «; = 1078, 6. = 107%, and A = 0.6 were used for 
Phase-1, and €; = 1079, e7 = 10~®, and A = 0.99, for Phase-2, and M = 1011, 

Codes FPS and FPSP were developed using MINOS 5.51 as a platform. 
Therefore, the three codes shared such features as preprocessing, scaling, LUSOL 
etc. Only the Mi50lp module was replaced by programs written by the author 
himself, with minor changes in few other modules. All codes used the default 
options. 

Compiled using Visual Fortran 5.0, all codes were run under a Microsoft Win- 
dows XP Professional version 2002 on an ACER PC with an Intel(R) Pentium(R) 4 
CPU 3.06GHz, 1.00GB memory, and about 16 digits of precision. 

The first set of test problems included all 16 problems from Kennington and 
the second included all 17 problems from BPMPD that were more than 500KB in 
compressed form (Appendix C: Tables C.2, C.3). 

In the tables, test problems are ordered by their sizes, in terms of m+n as before, 
where m and n are the numbers of rows and columns of the constraint matrix, 
excluding slack variables. All reported CPU times were measured in seconds with 
utility routine CPU_TIME, excluding the time spent on preprocessing and scaling. 
FPS and FPSP used 10~° as the equality tolerance (for holding of Ax = b), 
compared with MINOS, which used 10-4, as usual. 

Numerical results obtained with Kennington and BPMPD problems are dis- 
played, respectively, in Tables F.1 and F.2, where the total iterations and time 
required for solving each problem are listed in the columns labeled Itns and Time 
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Table F.1 Statistics for 16 Kennington problems 


Problem 
KEN-07 
CRE-C 
CRE-A 
PDS-02 
OSA-07 
KEN-11 
PDS-06 
OSA-14 
PDS-10 
KEN-13 
CRE-D 
CRE-B 
OSA-30 
PDS-20 
OSA-60 
KEN-18 
Total 


MINOS FPSP FPS 
Tins 


suo | 905 | um | 141005 | 
35162 
64771 
154801 
304313 


Table F.2 Statistics for 17 BPMPD problems 


Problem 


RAT7A 
NSCT1 
NSCT2 


RIUTING 


DBIR1 
DBIR2 


T0331-4L 


MINOS—_‘[FPSP FPS 
Tins 
108575 


NEMSEMM2 
SOUTHERN 
WORLD.MO 


WORLD 


2627208 61340.78 | 1260355 27315.31 | 165045 


NEMSEMM1 


NW14 
LPLI 
DBIC1 
Total 


RADIO.PR 
RADIO.DU 


| 407 | 845 | 899 =| 19.20 | 690_| 
375734 
886208 
623810 
3 | 0.09 | 5055 | 56.11 | 5018 


Time 
2.02 
4.66 
8.09 
1.61 
5.02 
113.59 
40.36 
16.19 
75.47 
627.05 
13.23 
17.91 
49.33 
1643.14 
199.36 
19253.55 
22070.58 


Time 


52.03 
1.44 
1.45 

780.00 
0.61 
235.72 
1328.14 
58.00 
314.16 
2929.00 
4132.83 
253.64 
14.42 
104.41 
18679.69 


28885 .54 


14009.16 
55.66 
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Table F.3. Rario for 16 Kennington problems 


FPSPIFPS 
Problem tins | Time 
KEN.O7 0 | 1.0 
cREC [3069 [3678 | 0s [os | 15 | 15 [27 [26 
CREA [3517__[4067 | 05 [os | 10 | 09 | 22 | 21 
PDS.02 li | 
osao7 [ni9 (23949 | 09 [09 | 15 | 14 [16 | 16 
KEN-I1 0 | 1.0 
PDS-06 14 [14 
OSA-14 20 | 20 
PDS-10 16 | 16 
KEN-13 Lo | 1.0 
CRED 245 [27.1 
CREB 7.7 [24.1 
OSA-30 21 | 2A 
PDS-20 14 [14 
OSA-60 23 [23 
KEN-18 10 | 1.0 
avei6) |---| 66 | 32 | 90 | 35 [ta | 4 


under MINOS, FPSP, and FPS, separately. As MINOS solved problems RADIO.PR 
and RADIO.DU in exceptional few iterations. Results for the two problems are 
listed separately in the bottom lines and excluded from comparison with BPMPD 
set. 

In Table F.3, a performance comparison between the three codes with Kennington 
set is made by giving iteration and time ratios of MINOS to FPSP, MINOS to FPS, 
and FPSP to FPS for each problem. It is seen from the bottom line that FPSP and 
FPS outperformed MINOS remarkably, with average iterations ratios 6.6 and 9.0, 
and time ratios 3.2 and 3.5! It is noted that the difference between FPSP and FPS is 
small (with iterations ratio 1.4 and time ratio 1.1). 

In Table F.4, a comparison between the three codes with BPMPD set is made. 
It is seen from the second bottom line that FPSP and FPS outperformed MINOS, 
by average iterations ratios 2.8 and 9.5, and time ratios 3.4 and 9.0! The margin 
between FPSP and FPS is smaller (with iterations ratio 3.3 and time ratio 2.6). 

In the bottom line of Table F.4, listed are average ratios for the entire set of the 31 
test problems. FPSP and FPS again defeated MINOS: the average iterations ratios 
are 3.3 and 9.4, while the time ratios are 3.4 and 6.6. 

In summary, the simplex interior-point algorithm is significantly superior to the 
standard simplex algorithm, at least with the tested problems. 
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Table F.4 Ratio for 15 BPMPD problems 


MINOS/FPSP MINOS/FPS FPSP/FPS 
Problem |M___|N tins | Time 


RATTA 1.0 
NSCTI 138.9 [137.1 
NSCT2 130.1 125.7 
RIUTI 1.0 
DBIRI 3759/3918 
DBIR? 1.0 
Toss! [66s [aomis_[ 06 [06 | 06 | 06 | 10 | 10 
NEM2 2 
SOUTH 1.0 
WORM 6.6 
WORLD 6.6 
Nem. [3946 [71413 [09 [10 | 09 | 10 | 10 | 10 
Nwia [74 [123409 [os [oa | 06 | 06 | 13 | 13 
LPL 1 
DBICI 1.0 
wey |= |- | 28 [34 | 9s | 90 | 33 | 26 
wean |= |- 133 [34 | 94 | 66 | 28 | 20 
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